Function Repository Resource:

MonitorProgress

Source Notebook

Monitor the progress of a computation and display the current step and estimated time remaining

Contributed by: Lukas Lang

ResourceFunction["MonitorProgress"][expr]

attempts to monitor the progress of the evaluation of expr.

ResourceFunction["MonitorProgress"][expr,steps]

monitors the evaluation of expr, with manually indicated steps.

ResourceFunction["MonitorProgress"][operator][args]

monitors the progress of operator[args].

Dataset[][ResourceFunction["MonitorProgress"][Query],query,]

monitors the progress of the dataset query query.

Details and Options

ResourceFunction["MonitorProgress"] will estimate the remaining time based on the number of remaining steps and the time taken so far.
ResourceFunction["MonitorProgress"] can automatically monitor the progress of a wide variety of functions:
For the *By-type function, ResourceFunction["MonitorProgress"] will monitor only the progress of applying the function to the individual items. Anything else, e.g. the actual sorting, is not included.
For the functions listed in "operator forms", ResourceFunction["MonitorProgress"] will return an operator if the expression was an operator form.
For some types of expressions, ResourceFunction["MonitorProgress"] cannot determine the number of expected steps. In these cases, an "endless" progress bar is used.
ResourceFunction["MonitorProgress"] accepts the following options:
"Resolution"Automaticresolution with which to update the progress bar
"Timing"Truewhether to include timing information
"Label"Nonelabel for the progress indicator
"CurrentDisplayFunction"Fullhow to display the current item of the computation
ParallelizationAutomatichow to handle parallelization of the evaluation
The option "Resolution" can be set to the following:
stepsupdate the progress bar after the specified number of steps
Scaled[frac]update after a fraction of the total number of steps
Automaticuse an automatically determined setting
The setting "Resolution"Automatic is equivalent to Scaled[1/20] when the number of steps is known, and 5 otherwise.
The option "Timing" can be set to the following:
Trueinclude timing information
Falsedo not include timing information
Automaticonly include timing information if the total number of steps is known
The timing information provided includes time elapsed, time per step, estimated time remaining and estimated total time. The latter two are missing if the number of steps is unknown.
The option "CurrentDisplayFunction" can be set to the following:
Fullshow a short version of the current item, with a tooltip of the full item
Automaticdo not include a tooltip
Nonedo not display the current item
funcapply func before generating the tooltipped short version of the current item
{func}same as func, but do not include a tooltip
{func,toolfunc}apply toolfunc to generate the content of the tooltip
{func,Full}show the full current item in the tooltip
The function specified by "CurrentDisplayFunction" is effectively applied to the value set by ResourceFunction["MonitorProgress"]["SetCurrent"]/ResourceFunction["MonitorProgress"]["SetCurrentBy"].
ResourceFunction["MonitorProgress"] supports parallelized evaluation, with the progress being properly synchronized between kernels. The exact behavior can be specified using the Parallelization option:
Automaticonly use parallelization if the argument is explicitly parallelized
Falsedisable parallelization support
{opts}attempt parallelization using Parallelize with the specified options
Fullattempt parallelization using Parallelize; equivalent to {}
If ParallelizationAutomatic is specified, any parallelization options specified in expr are taken into account.
If ResourceFunction["MonitorProgress"] cannot automatically determine how to inject tracking into the evaluated expression, the necessary information can be provided manually. Apart from supplying the number of steps expected, it is also necessary to indicate when each step of the computation is finished. It is also possible to set a label for the currently processed item/step. The following functions can be used for this purpose:
ResourceFunction["MonitorProgress"]["Step"][args]indicates that one step is done; returns args
ResourceFunction["MonitorProgress"]["SetCurrent"][label]set the label of the current step to label; returns label
ResourceFunction["MonitorProgress"]["SetCurrentBy"][func][expr]set the label of the current step to func[expr]; returns expr
All three functions for manual monitoring return their arguments to allow for simple chaining.
If the number of steps is unknown, Indeterminate can be specified for steps instead. This causes an "endless" progress bar to be used.
ResourceFunction["MonitorProgress"] supports advanced evaluation control and will work nicely with Unevaluated arguments and functions with HoldAll and related attributes.
ResourceFunction["MonitorProgress"] will accurately predict the number of steps needed for mapping operations on SparseArray expressions.
Even though ResourceFunction["MonitorProgress"] is designed to incur minimal overhead, the performance impact might not be negligible if the individual steps are very fast. Furthermore, ResourceFunction["MonitorProgress"] might prevent automatic compilation performed by some functions.

Examples

Basic Examples (3) 

Monitor the progress of a Map operation:

In[1]:=
ResourceFunction["MonitorProgress"]@Map[(Pause@0.2; f@#) &, Range@10]
Out[1]=
Out[1]=

Monitor the progress of a 2D Table generation:

In[2]:=
ResourceFunction["MonitorProgress"]@Table[
  Pause@RandomReal@{0.2, 0.4}; 2^i + 3^j,
  {i, 5}, {j, 3}
  ]
Out[2]=
Out[2]=

In some cases, the total number of steps is not known:

In[3]:=
list = {};
ResourceFunction["MonitorProgress"]@Do[
   Pause[0.05 RandomReal[]];
   If[Mean@AppendTo[list, i^2] > 1000, Break[]],
   {i, \[Infinity]}
   ];
list
Out[3]=
Out[5]=

Scope (6) 

Create an operator for later use:

In[6]:=
myOp = ResourceFunction["MonitorProgress"][Map[(Pause@0.2; f@#) &]]
Out[6]=

Apply the operator to a list:

In[7]:=
myOp@Range@10
Out[7]=
Out[7]=

Indicate steps manually for cases where they cannot be automatically inferred:

In[8]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/40abac96-fc81-4835-8892-4d1505cdf3b3"]
Out[8]=

MonitorProgress supports advanced evaluation control and does not evaluate any arguments to functions with Hold attributes:

In[9]:=
Attributes[myFunc] = {HoldAll};
myFunc[arg_] := (Pause@1; Hold@arg)
ResourceFunction["MonitorProgress"][
 myFunc /@ Unevaluated@{Print@1, Print@2, Print@3}]
Out[9]=
Out[11]=

Mapping operations on SparseArray expressions are also supported, and the correct number of steps is determined:

In[12]:=
ResourceFunction[
 "MonitorProgress"][(Pause@1; f@#) & /@ SparseArray[{2 -> 1}, 4]]
Out[12]=
Out[12]=

MonitorProgress will attempt to construct tracked operators out of Query expressions:

In[13]:=
myQuery = ResourceFunction["MonitorProgress"]@
  Query[All, {1 -> ((Pause@0.5; f@#) &)}]
Out[13]=
In[14]:=
myQuery@Dataset@Table[{i, j}, {i, 3}, {j, 3}]
Out[14]=
Out[14]=

Monitor a Dataset query without having to wrap the whole argument explicitly in MonitorProgress@Query[]:

In[15]:=
Dataset[Table[{i, j}, {i, 3}, {j, 3}]][
 ResourceFunction["MonitorProgress"]@Query, All, (Pause@0.5; f@#) &]
Out[15]=
Out[15]=

Options (18) 

CurrentDisplayFunction (7) 

With the default setting "CurrentDisplayFunction"Full, a short version of the current item is displayed with a tooltip of the full item:

In[16]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}]];
Out[16]=

Do not include the tooltip:

In[17]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> Automatic];
Out[17]=

Do not display the current item at all (this can improve performance for complex items):

In[18]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> None];
Out[18]=

Use a custom function to generate the expression to display:

In[19]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> (Style[#, Red] &)];
Out[19]=

Use a custom function to generate the expression to display, but do not include a tooltip:

In[20]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &}];
Out[20]=

Use a different function to generate the tooltip content:

In[21]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &, Style[#, Blue] &}];
Out[21]=

Show the full, unmodified current item in the tooltip:

In[22]:=
ResourceFunction["MonitorProgress"][
  Pause@0.5 & /@ Table[i*j, {j, 10}, {i, 10}], "CurrentDisplayFunction" -> {Style[#, Red] &, Full}];
Out[22]=

Label (2) 

With the default setting "Label"None, no label is included:

In[23]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}]
Out[23]=
Out[23]=

Set a custom label for the progress indicator:

In[24]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}, "Label" -> "My label"]
Out[24]=
Out[25]=

Parallelization (3) 

With the default setting ParallelizationAutomatic, parallelization support is only enabled for explicitly parallelized computations:

In[26]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20]]
Out[26]=
In[27]:=
ResourceFunction["MonitorProgress"][Parallelize@Table[$KernelID, 20]]
Out[27]=

Attempt explicit parallelization using the specified options:

In[28]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20], Parallelization -> {Method -> "FinestGrained"}]
Out[28]=

This is equivalent to explicitly wrapping the argument in Parallelize[,opts]:

In[29]:=
ResourceFunction["MonitorProgress"][
 Parallelize[Table[$KernelID, 20], Method -> "FinestGrained"]]
Out[29]=

Attempt parallelization using the default settings:

In[30]:=
ResourceFunction["MonitorProgress"][Table[$KernelID, 20], Parallelization -> Full]
Out[30]=

Resolution (3) 

With the default setting "Resolution"Automatic, the progress bar is updated 20 times:

In[31]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100];
Out[31]=

Update the progress bar only five times:

In[32]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100, "Resolution" -> Scaled[1/5]];
Out[32]=

Update the progress bar after every single step:

In[33]:=
ResourceFunction[
  "MonitorProgress"][(Pause[RandomReal[] 0.05]; f@#) & /@ Range@100, "Resolution" -> 1];
Out[33]=

Timing (3) 

With the default setting "Timing"True, timing information is included in all progress indicators:

In[34]:=
ResourceFunction["MonitorProgress"][
 Do[Pause@0.1; If[i > 30, Break[]], {i, \[Infinity]}]]
Out[34]=

Use "Timing" Automatic to enable timing only when the total number of steps is known:

In[35]:=
ResourceFunction["MonitorProgress"][
 Do[Pause@0.1; If[i > 30, Break[]], {i, \[Infinity]}], "Timing" -> Automatic]
Out[35]=

Disable timing in all cases:

In[36]:=
ResourceFunction["MonitorProgress"][Pause /@ {1, 1, 1}, "Timing" -> False]
Out[36]=
Out[37]=

Possible Issues (7) 

MonitorProgress computes the total time needed based on the average of all previous steps, which leads to inaccurate predictions for highly variable step durations:

In[38]:=
ResourceFunction["MonitorProgress"][Table[Pause[0.1 i], {i, 10}]];
Out[38]=

Timing information can be disabled in these cases:

In[39]:=
ResourceFunction["MonitorProgress"][Table[Pause[0.1 i], {i, 10}], "Timing" -> False];
Out[39]=

Although MonitorProgress will return an operator for arbitrary Function expressions, the resulting expression must still be supported by MonitorProgress:

In[40]:=
myOp = ResourceFunction["MonitorProgress"][func[#, "some Arg", 2] &]
Out[40]=
In[41]:=
myOp[3]
Out[41]=

MonitorProgress does not support all types of Query expressions:

In[42]:=
ResourceFunction["MonitorProgress"][Query[2 ;; 3, myFunc]]
Out[42]=

MonitorProgress tracks queries by effectively applying Normal to them. This means that certain special behaviors of Query may not be faithfully reproduced:

In[43]:=
myRawQuery = Query[All, Mean];
myQuery = ResourceFunction["MonitorProgress"]@Evaluate@myRawQuery;
data = Dataset@
  Table[If[i == j, Missing["NotAvailable"], i j], {i, 3}, {j, 3}]
Out[16]=
In[44]:=
myRawQuery@data
Out[44]=
In[45]:=
myQuery@data
Out[45]=

Although the overhead of MonitorProgress is low, it might not be negligible in all cases:

In[46]:=
withMonitorProgress = Table[
   First@AbsoluteTiming@ResourceFunction["MonitorProgress"][Table[
       Pause[5^-n]; RandomReal[],
       {i, 5^n}
       ]
      ],
   {n, 5}
   ];
withoutMonitorProgress = Table[
   First@AbsoluteTiming[Table[
      Pause[5^-n]; RandomReal[],
      {i, 5^n}
      ]
     ],
   {n, 5}
   ];
In[47]:=
ListLinePlot[{withoutMonitorProgress, withMonitorProgress}, PlotLegends -> {"Without MonitorProgress", "With MonitorProgress"}, PlotRange -> All]
Out[47]=

Very extreme cases of slowdown occur when the untracked operation could be automatically compiled, since the tracking prevents this auto-compilation:

In[48]:=
ResourceFunction["MonitorProgress"]@
   Table[i^2, {i, 1000000}]; // AbsoluteTiming
Out[48]=
In[49]:=
Table[i^2, {i, 1000000}]; // AbsoluteTiming
Out[49]=

Parallelization might fail if the underlying operation cannot be parallelized:

In[50]:=
ResourceFunction["MonitorProgress"][BlockMap[f, Range@10, 2], Parallelization -> Full]
Out[50]=

Publisher

Lukas Lang

Version History

  • 2.2.1 – 15 March 2023
  • 2.2.0 – 16 August 2022
  • 2.1.2 – 09 August 2022
  • 2.1.1 – 18 July 2022
  • 2.1.0 – 05 July 2022
  • 2.0.0 – 01 September 2020
  • 1.0.0 – 18 December 2019

Related Resources

Author Notes

Changes from version 2.2.0 to 2.2.1:

Fixed MapAt[…,<||>,…] type expressions not being processed properly

Changes from version 2.1.2 to 2.2.0:

Added support for many *By-type functions and their operator forms

Changes from version 2.1.1 to 2.1.2:

MonitorProgress now reports more fine-grained progress for multi-iterator Table and Do with iterators of the form {var,start,end,step}

Changes from version 2.1.0 to 2.1.1:

Fixed spacing in default current item display for Association

Changes from version 2.0.0 to 2.1.0:

Reduced overhead of MonitorProgress by off-loading as much as possible to the front-end helper kernel
Added minimal support for text-based interfaces
Added support for Apply
Added support for options in Dataset[][MonitorProgress[Query,opts],query] type dataset queries
Fixed issues with nested MonitorProgress calls where the inner call was determined to be 0 or 1 steps
Fixed issues when using TimingFalse and LabelNone simultaneously
MonitorProgress panels are now deleted when opening a notebook that was saved while a MonitorProgress panel was being displayed

Changes from version 1.0.0 to 2.0.0:

Fixed Parallelization not working

License Information