Function Repository Resource:

DistributeResourceFunctions

Source Notebook

Distribute resource function definitions to parallel kernels

Contributed by: Richard Hennigan (Wolfram Research)

ResourceFunction["DistributeResourceFunctions"][rf]

distributes metadata and definitions for the ResourceFunction given by rf to parallel kernels.

ResourceFunction["DistributeResourceFunctions"][{rf1,rf2,}]

distributes multiple resource functions.

ResourceFunction["DistributeResourceFunctions"][rfs,ker]

distributes only to the kernels specified in ker.

Details and Options

The value for rf can be any of the following:
"name"the name of a resource function
"uuid"the UUID of a resource function
ResourceObject[]a resource object with type "Function"
ResourceFunction[]a resource function
Allall resource functions loaded in the current session
ResourceFunction["DistributeResourceFunctions"] applies itself recursively to any resource functions found appearing in the definition of rf.
ResourceFunction["DistributeResourceFunctions"] is similar to DistributeDefinitions, but it also preloads ResourceObject metadata in each parallel kernel so they do not need to read the resource from disk.
ResourceFunction["DistributeResourceFunctions"] will automatically launch parallel kernels if necessary.
New kernels launched after distributing resource functions will not automatically have resource functions distributed.
If a resource function has already been distributed to all kernels, it is not distributed again.
ResourceFunction["DistributeResourceFunctions"][rf] is effectively equivalent to ResourceFunction["DistributeResourceFunctions"][rf,Kernels[]].

Examples

Basic Examples (3) 

Distribute resource function definitions to parallel kernels:

In[1]:=
ResourceFunction["DistributeResourceFunctions"]["ColorToHex"]
Out[1]=

Now the function can be used quickly on each kernel:

In[2]:=
ParallelEvaluate[AbsoluteTiming[ResourceFunction["ColorToHex"][Red]]]
Out[2]=

Compare to a function that has not been distributed:

In[3]:=
ParallelEvaluate[
 AbsoluteTiming[ResourceFunction["HashHue"]["Hello World"]]]
Out[3]=

Scope (4) 

Distribute multiple resource functions at once:

In[4]:=
ResourceFunction[
 "DistributeResourceFunctions"][{"BirdSay", "Bob", "Wiggled"}]
Out[4]=

Using a resource function loads it into the current session:

In[5]:=
ResourceFunction["SpeechBubble"][
 ResourceFunction["PartyParrot"][
  "Birdnardo"], "I only load the best resource functions"]
Out[5]=

Distribute all currently loaded resource functions to parallel kernels:

In[6]:=
ResourceFunction["DistributeResourceFunctions"][All]
Out[6]=

Only functions that have not been distributed yet will be sent to parallel kernels:

In[7]:=
ResourceFunction[
 "DistributeResourceFunctions"][{"WoodFramed", "BirdSay", "Spookify"}]
Out[7]=
In[8]:=
ResourceFunction[
 "DistributeResourceFunctions"][{"WoodFramed", "BirdSay", "Spookify"}]
Out[8]=

Distribute to specific kernels:

In[9]:=
ResourceFunction[
 "DistributeResourceFunctions"]["WolfieSay", {1, 2, 3, 4}]
Out[9]=

Compare initial timing with other kernels:

In[10]:=
ParallelEvaluate[$KernelID -> First[AbsoluteTiming[ResourceFunction["WolfieSay"]["test"]]]]
Out[10]=
In[11]:=
BarChart[Association@%, ChartLabels -> Automatic, ScalingFunctions -> "Log"]
Out[11]=

Properties and Relations (4) 

Normally, initial loading time for resource functions increases with the number of parallel kernels, because each kernel must read the definition sequentially:

In[12]:=
LocalCache[DefineResourceFunction[# + 1 &, "AddOne"]]
Out[12]=
In[13]:=
ParallelMap[AbsoluteTiming[ResourceFunction["AddOne"][#]] &, Range[20]]
Out[13]=

The number of slow evaluations will correspond to the number of kernels:

In[14]:=
Count[%, {t_, _} /; t > .05]
Out[14]=
In[15]:=
$KernelCount
Out[15]=

Distribute a local resource function to parallel kernels to bypass initial loading time:

In[16]:=
LocalCache[DefineResourceFunction[# + 2 &, "AddTwo"]]
Out[16]=
In[17]:=
ResourceFunction["DistributeResourceFunctions"]["AddTwo"]
Out[17]=

Now there is no initial loading overhead:

In[18]:=
ParallelMap[AbsoluteTiming[ResourceFunction["AddTwo"][#]] &, Range[20]]
Out[18]=
In[19]:=
Count[%, {t_, _} /; t > .05]
Out[19]=

Version History

  • 1.1.0 – 29 March 2022
  • 1.0.0 – 06 November 2020

Related Resources

License Information