Function Repository Resource:

IncrementalParallelSelect

Source Notebook

Incrementally select elements in parallel from a list or infinite range that satisfy a predicate

Contributed by: Roman E. Maeder

ResourceFunction["IncrementalParallelSelect"][list,crit]

computes Select[list,crit] in parallel and returns partial results when aborted.

ResourceFunction["IncrementalParallelSelect"][n,crit]

computes Select[Range[n,Infinity],crit] in parallel until aborted.

Details and Options

A computation can be aborted with EvalutationAbortEvaluation or with the stop button in the progress display.
Using the stop button lets pending evaluations run to completion; an abort terminates immediately.
When a computation is aborted, the results found so far are returned.
ResourceFunction["IncrementalParallelSelect"][n,crit] runs until aborted, computing crit[n], crit[n+1], …, with no upper limit.
ResourceFunction["IncrementalParallelSelect"] takes the DistributedContexts option like other parallel functions.
The temporary progress display shows the number of kernels used, the number of elements already tested, and the number of elements for which crit[e] is True. It also shows the elapsed time and the average time for each computation.

Examples

Basic Examples (2) 

Search for Mersenne primes in the range 1 to 1200:

In[1]:=
ResourceFunction["IncrementalParallelSelect"][Range[1200], PrimeQ[2^Prime[#] - 1] &]
Out[1]=

Search for Mersenne primes with no upper limit, until aborted:

In[2]:=
ResourceFunction["IncrementalParallelSelect"][1, PrimeQ[2^Prime[#] - 1] &]
Out[2]=

Scope (2) 

The "Results" opener in the temporary progress display allows one to see the results as they are found:

In[3]:=
ResourceFunction["IncrementalParallelSelect"][Range[1200], PrimeQ[2^Prime[#] - 1] &]
Out[3]=

When aborting a computation with the stop button, the number of evaluations that are still running is displayed:

In[4]:=
ResourceFunction["IncrementalParallelSelect"][2200, PrimeQ[2^Prime[#] - 1] &]
Out[4]=

Aborting with EvalutationAbortEvaluation terminates more quickly:

In[5]:=
ResourceFunction["IncrementalParallelSelect"][1200, PrimeQ[2^Prime[#] - 1] &]
Out[5]=

Options (2) 

DistributedContexts (2) 

By default, definitions in the current context are distributed automatically:

In[6]:=
remote[_] := (Pause[0.1]; PrimeQ[$KernelID])
In[7]:=
ResourceFunction["IncrementalParallelSelect"][Range[20], remote]
Out[7]=

Do not distribute any definitions of functions. In this case, the definition of local is applied only after the results are returned to the local kernel:

In[8]:=
local[x_] := (Pause[0.1]; PrimeQ[$KernelID])
In[9]:=
ResourceFunction["IncrementalParallelSelect"][Range[20], local, DistributedContexts -> None]
Out[9]=

Properties and Relations (2) 

IncrementalParallelSelect returns the same result as Select, or its parallelized, non-incremental version:

In[10]:=
Parallelize[Select[Range[1200], PrimeQ[2^Prime[#] - 1] &]]
Out[10]=

The standard parallel version cannot return partial results when aborted:

In[11]:=
Parallelize[Select[Range[1200], PrimeQ[2^Prime[#] - 1] &]]
Out[11]=

Neat Examples (1) 

Find memorable primes:

In[12]:=
memorableNumber[n_] := With[{seq = IntegerString /@ Range[n]},
  FromDigits@StringJoin@Join[seq, Rest[Reverse[seq]]]
  ]
In[13]:=
memorableNumber[10]
Out[13]=
In[14]:=
ResourceFunction["IncrementalParallelSelect"][1, PrimeQ@*memorableNumber]
Out[14]=
In[15]:=
Short[memorableNumber[2446]]
Out[15]=

Publisher

Roman Maeder

Version History

  • 1.0.0 – 11 January 2022

Related Resources

Author Notes

IncrementalParallelSelect is best for experiments where each evaluation takes a long time, or the extent of the input range is not yet known.

For long input ranges and short evaluation times, Parallelize[Select[…]] incurs less overhead and may be faster.

License Information