Wolfram Research

Function Repository Resource:

MonitorProgress

Source Notebook

Monitor the progress of a computation and display the current step and estimated time remaining

Contributed by: Lukas Lang

ResourceFunction["MonitorProgress"][expr]

attempts to monitor the progress of the evaluation of expr.

ResourceFunction["MonitorProgress"][expr,steps]

monitors the evaluation of expr, with manually indicated steps.

ResourceFunction["MonitorProgress"][operator][args]

monitors the progress of operator[args].

Dataset[][ResourceFunction["MonitorProgress"][Query],query,]

monitors the progress of the dataset query query.

Details and Options

ResourceFunction["MonitorProgress"] will estimate the remaining time based on the number of remaining steps and the time taken so far.
ResourceFunction["MonitorProgress"] can automatically monitor the progress of a wide variety of functions:
For the functions listed in "operator forms", ResourceFunction["MonitorProgress"] will return an operator if the expression was an operator form.
For some types of expressions, ResourceFunction["MonitorProgress"] cannot determine the number of expected steps. In these cases, an "endless" progress bar is used.
ResourceFunction["MonitorProgress"] accepts the following options:
"Resolution" Automatic resolution with which to update the progress bar
"Timing" True whether to include timing information
"Label" None label for the progress indicator
"CurrentDisplayFunction" Full how to display the current item of the computation
Parallelization Automatic how to handle parallelization of the evaluation
The option "Resolution" can be set to the following:
steps update the progress bar after the specified number of steps
Scaled [ frac ] update after a fraction of the total number of steps
Automatic use an automatically determined setting
The setting "Resolution"Automatic is equivalent to Scaled[1/20] when the number of steps is known, and 5 otherwise.
The option "Timing" can be set to the following:
True include timing information
False do not include timing information
Automatic only include timing information if the total number of steps is known
The timing information provided includes time elapsed, time per step, estimated time remaining and estimated total time. The latter two are missing if the number of steps is unknown.
The option "CurrentDisplayFunction" can be set to the following:
Full show a short version of the current item, with a tooltip of the full item
Automatic do not include a tooltip
None do not display the current item
func apply func before generating the tooltipped short version of the current item
{ func } same as func, but do not include a tooltip
{ func , toolfunc } apply toolfunc to generate the content of the tooltip
{ func , Full } show the full current item in the tooltip
The function specified by "CurrentDisplayFunction" is effectively applied to the value set by ResourceFunction["MonitorProgress"]["SetCurrent"]/ResourceFunction["MonitorProgress"]["SetCurrentBy"].
ResourceFunction["MonitorProgress"] supports parallelized evaluation, with the progress being properly synchronized between kernels. The exact behavior can be specified using the Parallelization option:
Automatic only use parallelization if the argument is explicitly parallelized
False disable parallelization support
{ opts } attempt parallelization using Parallelize with the specified options
Full attempt parallelization using Parallelize; equivalent to {}
If ParallelizationAutomatic is specified, any parallelization options specified in expr are taken into account.
If ResourceFunction["MonitorProgress"] cannot automatically determine how to inject tracking into the evaluated expression, the necessary information can be provided manually. Apart from supplying the number of steps expected, it is also necessary to indicate when each step of the computation is finished. It is also possible to set a label for the currently processed item/step. The following functions can be used for this purpose:
ResourceFunction["MonitorProgress"]["Step"] [ args ] indicates that one step is done; returns args
ResourceFunction["MonitorProgress"]["SetCurrent"] [ label ] set the label of the current step to label; returns label
ResourceFunction["MonitorProgress"]["SetCurrentBy"][func][expr] set the label of the current step to func[expr]; returns expr
All three functions for manual monitoring return their arguments to allow for simple chaining.
If the number of steps is unknown, Indeterminate can be specified for steps instead. This causes an "endless" progress bar to be used.
ResourceFunction["MonitorProgress"] supports advanced evaluation control and will work nicely with Unevaluated arguments and functions with HoldAll and related attributes.
ResourceFunction["MonitorProgress"] will accurately predict the number of steps needed for mapping operations on SparseArray expressions.
Even though ResourceFunction["MonitorProgress"] is designed to incur minimal overhead, the performance impact might not be negligible if the individual steps are very fast. Furthermore, ResourceFunction["MonitorProgress"] might prevent automatic compilation performed by some functions.

Examples

Basic Examples (3) 

Monitor the progress of a Map operation:

In[1]:=
ResourceFunction["MonitorProgress"]@Map[(Pause@0.2; f@#) &, Range@10]
Out[1]=
Out[1]=

Monitor the progress of a 2D Table generation:

In[2]:=
ResourceFunction["MonitorProgress"]@Table[
  Pause@RandomReal@{0.2, 0.4}; 2^i + 3^j,
  {i, 5}, {j, 3}
  ]
Out[2]=
Out[2]=

In some cases, the total number of steps is not known:

In[3]:=
list = {};
ResourceFunction["MonitorProgress"]@Do[
   Pause[0.05 RandomReal[]];
   If[Mean@AppendTo[list, i^2] > 1000, Break[]],
   {i, \[Infinity]}
   ];
list
Out[3]=
Out[5]=

Scope (6) 

Create an operator for later use:

In[6]:=
myOp = ResourceFunction["MonitorProgress"][Map[(Pause@0.2; f@#) &]]
Out[6]=

Apply the operator to a list:

In[7]:=
myOp@Range@10
Out[7]=
Out[7]=

Indicate steps manually for cases where they cannot be automatically inferred:

In[8]:=
ResourceFunction["MonitorProgress"][
 ResourceFunction["MonitorProgress"]["SetCurrent"]["Step 1"];
 Pause@1;
 ResourceFunction["MonitorProgress"]["Step"][];
 ResourceFunction["MonitorProgress"]["SetCurrent"]["Step 2"];
 Pause@1;
 ResourceFunction["MonitorProgress"]["Step"][];
 ResourceFunction["MonitorProgress"]["SetCurrent"]["Step 3"];
 Pause@1;
 ResourceFunction["MonitorProgress"]["Step"][];,
 3
 ]
Out[8]=

MonitorProgress supports advanced evaluation control and does not evaluate any arguments to functions with Hold attributes:

In[9]:=
Attributes[myFunc] = {HoldAll};
myFunc[arg_] := (Pause@1; Hold@arg)
ResourceFunction["MonitorProgress"][
 myFunc /@ Unevaluated@{Print@1, Print@2, Print@3}]
Out[9]=
Out[11]=

Mapping operations on SparseArray expressions are also supported, and the correct number of steps is determined:

In[12]:=
ResourceFunction[
 "MonitorProgress"][(Pause@1; f@#) & /@ SparseArray[{2 -> 1}, 4]]
Out[12]=
Out[12]=

MonitorProgress will attempt to construct tracked operators out of Query expressions:

In[13]:=
myQuery = ResourceFunction["MonitorProgress"]@
  Query[All, {1 -> ((Pause@0.5; f@#) &)}]
Out[13]=
In[14]:=
myQuery@Dataset@Table[{i, j}, {i, 3}, {j, 3}]
Out[14]=
Out[14]=

Monitor a Dataset query without having to wrap the whole argument explicitly in MonitorProgress@Query[]:

In[15]:=
Dataset[Table[{i, j}, {i, 3}, {j, 3}]][
 ResourceFunction["MonitorProgress"]@Query, All, (Pause@0.5; f@#) &]
Out[15]=
Out[15]=

Options (18) 

CurrentDisplayFunction (7) 

With the default setting "CurrentDisplayFunction"Full, a short version of the current item is displayed with a tooltip of the full item:

In[16]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}]];
Out[16]=

Do not include the tooltip:

In[17]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> Automatic];
Out[17]=

Do not display the current item at all (this can improve performance for complex items):

In[18]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> None];
Out[18]=

Use a custom function to generate the expression to display:

In[19]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> (Style[#, Red] &)];
Out[19]=

Use a custom function to generate the expression to display, but do not include a tooltip:

In[20]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &}];
Out[20]=

Use a different function to generate the tooltip content:

In[21]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &, Style[#, Blue] &}];
Out[21]=

Show the full, unmodified current item in the tooltip:

In[22]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &, Full}];
Out[22]=

Label (2) 

With the default setting "Label"None, no label is included:

In[23]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}]
Out[23]=
Out[23]=

Set a custom label for the progress indicator:

In[24]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}, "Label" -> "My label"]
Out[24]=
Out[25]=

Parallelization (3) 

With the default setting ParallelizationAutomatic, parallelization support is only enabled for explicitly parallelized computations:

In[26]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20]]
Out[26]=
In[27]:=
ResourceFunction["MonitorProgress"][Parallelize@Table[$KernelID, 20]]
Out[27]=

Attempt explicit parallelization using the specified options:

In[28]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20], Parallelization -> {Method -> "FinestGrained"}]
Out[28]=

This is equivalent to explicitly wrapping the argument in Parallelize[,opts]:

In[29]:=
ResourceFunction["MonitorProgress"][
 Parallelize[Table[$KernelID, 20], Method -> "FinestGrained"]]
Out[29]=

Attempt parallelization using the default settings:

In[30]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20], Parallelization -> Full]
Out[30]=

Resolution (3) 

With the default setting "Resolution"Automatic, the progress bar is updated 20 times:

In[31]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100];
Out[31]=

Update the progress bar only five times:

In[32]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100, "Resolution" -> Scaled[1/5]];
Out[32]=

Update the progress bar after every single step:

In[33]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100, "Resolution" -> 1];
Out[33]=

Timing (3) 

With the default setting "Timing"True, timing information is included in all progress indicators:

In[34]:=
ResourceFunction["MonitorProgress"][
 Do[Pause@0.1; If[i > 30, Break[]], {i, \[Infinity]}]]
Out[34]=

Use "Timing" Automatic to enable timing only when the total number of steps is known:

In[35]:=
ResourceFunction["MonitorProgress"][
 Do[Pause@0.1; If[i > 30, Break[]], {i, \[Infinity]}], "Timing" -> Automatic]
Out[35]=

Disable timing in all cases:

In[36]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}, "Timing" -> False]
Out[36]=
Out[37]=

Possible Issues (7) 

MonitorProgress computes the total time needed based on the average of all previous steps, which leads to inaccurate predictions for highly variable step durations:

In[38]:=
ResourceFunction["MonitorProgress"][Table[Pause[0.1 i], {i, 10}]];
Out[38]=

Timing information can be disabled in these cases:

In[39]:=
ResourceFunction["MonitorProgress"][Table[Pause[0.1 i], {i, 10}], "Timing" -> False];
Out[39]=

Although MonitorProgress will return an operator for arbitrary Function expressions, the resulting expression must still be supported by MonitorProgress:

In[40]:=
myOp = ResourceFunction["MonitorProgress"][func[#, "some Arg", 2] &]
Out[40]=
In[41]:=
myOp[3]
Out[41]=

MonitorProgress does not support all types of Query expressions:

In[42]:=
ResourceFunction["MonitorProgress"][Query[2 ;; 3, myFunc]]
Out[42]=

MonitorProgress tracks queries by effectively applying Normal to them. This means that certain special behaviors of Query may not be faithfully reproduced:

In[43]:=
myRawQuery = Query[All, Mean];
myQuery = ResourceFunction["MonitorProgress"]@Evaluate@myRawQuery;
data = Dataset@
  Table[If[i == j, Missing["NotAvailable"], i j], {i, 3}, {j, 3}]
Out[16]=
In[44]:=
myRawQuery@data
Out[44]=
In[45]:=
myQuery@data
Out[45]=

Although the overhead of MonitorProgress is low, it might not be negligible in all cases:

In[46]:=
withMonitorProgress = Table[
   First@AbsoluteTiming@ResourceFunction["MonitorProgress"][Table[
       Pause[5^-n]; RandomReal[],
       {i, 5^n}
       ]
      ],
   {n, 6}
   ];
withoutMonitorProgress = Table[
   First@AbsoluteTiming[Table[
      Pause[5^-n]; RandomReal[],
      {i, 5^n}
      ]
     ],
   {n, 6}
   ];
In[47]:=
ListLinePlot[{withoutMonitorProgress, withMonitorProgress}, PlotLegends -> {"Without MonitorProgress", "With MonitorProgress"}, PlotRange -> All]
Out[47]=

Very extreme cases of slowdown occur when the untracked operation could be automatically compiled, since the tracking prevents this auto-compilation:

In[48]:=
ResourceFunction["MonitorProgress"]@
   Table[i^2, {i, 1000000}]; // AbsoluteTiming
Out[48]=
In[49]:=
Table[i^2, {i, 1000000}]; // AbsoluteTiming
Out[49]=

Parallelization might fail if the underlying operation cannot be parallelized:

In[50]:=
ResourceFunction["MonitorProgress"][BlockMap[f, Range@10, 2], Parallelization -> Full]
Out[50]=

Resource History

Related Resources

Author Notes

Changes from Version 1.0.0 to 1.1.0:

Fixed Parallelization not working

License Information