Function Repository Resource:

MapBatched

Source Notebook

Divide a Map operation into batches, performing an evaluation between batches

Contributed by: Joshua Schrier

ResourceFunction["MapBatched"][f,expr]

applies f to each element on the first level of expr,pausing for one second every five elements.

ResourceFunction["MapBatched"][f,expr,action]

applies f to each element on the first level of expr,performing action every five elements.

ResourceFunction["MapBatched"][f,expr,action,n]

uses batches of length n.

Details and Options

The default action is a one-second Pause and the default size for n is 5. This facilitates interacting with external web APIs that request limits on query frequency.
The Map operation occurs at level {1} only.

Examples

Basic Examples (2) 

MapBatched applies a function to a list or association. By default, batches consist of 5 entries, and the action taken is to Pause for one second per batch:

In[1]:=
ResourceFunction["MapBatched"][# &, Range[20]] // AbsoluteTiming
Out[1]=

The result using MapBatched identical to an ordinary Map except for inserting this Pause:

In[2]:=
Map[# &, Range[20]] // AbsoluteTiming
Out[2]=

When the number of elements is less than the batch size, no Pause is inserted:

In[3]:=
ResourceFunction["MapBatched"][# &, {1, 2, 3, 4, 5}] // AbsoluteTiming
Out[3]=

Scope (2) 

The expr can be any expression over which the function f can be mapped (e.g., lists and associations). The action can be any expression:

In[4]:=
ResourceFunction["MapBatched"][# &, Range[15], Print["batch action"]; Pause[0.2];]
Out[4]=

The size n can be any integer:

In[5]:=
ResourceFunction["MapBatched"][# &, Range[10], Print["batch action"],
  2]
Out[5]=

Applications (3) 

One intended usage is to access web APIs that restrict request rates. For example, manual access to PubChem via the PUG API:

In[6]:=
retrieveName[myName_String] := With[
   {pugrest = "https://pubchem.ncbi.nlm.nih.gov/rest/pug", pugoper = "property/CanonicalSMILES",
    pugout = "txt",
    pugin = "compound/name/" <> myName},
   URLExecute@URLBuild[{pugrest, pugin, pugoper, pugout}]];

Define a list of chemicals:

In[7]:=
names = {"water", "benzene", "methanol", "ethene", "ethanol", "propene", "1-propanol", "2-propanol", "butadiene", "1-butanol", "2-butanol", "tert-butanol"};

Retrieve results from the API in batches:

In[8]:=
ResourceFunction["MapBatched"][retrieveName, names]
Out[8]=

Publisher

Joshua Schrier

Version History

  • 1.0.0 – 16 January 2020

Related Resources

Author Notes

Joshua Schrier <jschrier@fordham.edu> https://scholar.google.com/citations?user=zJC_ 7roAAAAJ&hl=en Twitter: @Joshua Schrier github.com/jschrier

License Information