# Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

Monitor the progress of a computation and display the current step and estimated time remaining

Contributed by:
Lukas Lang

ResourceFunction["MonitorProgress"][ attempts to monitor the progress of the evaluation of | |

ResourceFunction["MonitorProgress"][ monitors the evaluation of | |

ResourceFunction["MonitorProgress"][ monitors the progress of | |

ResourceFunction["MonitorProgress"] will estimate the remaining time based on the number of remaining steps and the time taken so far.

ResourceFunction["MonitorProgress"] can automatically monitor the progress of a wide variety of functions:

map-type functions | Map,ParallelMap,AssociationMap,MapIndexed,MapAt, BlockMap,Scan,Apply |

queries | Query |

table-type functions | Table,ParallelTable,Do,ParallelDo,Array,ParallelArray |

recursion functions | Nest,NestList,Fold,FoldList,FoldPair,FoldPairList |

filtering/counting functions | Select,SelectFirst,LengthWhile,TakeWhile |

*By-type functions | CountDistinctBy,CountsBy,DeleteDuplicatesBy,GatherBy, GroupBy,KeySortBy,MaximalBy,MinimalBy,OrderingBy, ReverseSortBy,SortBy,SplitBy,TakeLargestBy,TakeSmallestBy |

procedural programming functions | For,While |

operator forms | Map,AssociationMap,MapIndexed,MapAt,Apply,Query, Fold,FoldList,Select,SelectFirst,CountDistinctBy,CountsBy, DeleteDuplicatesBy,GroupBy,KeySortBy,MaximalBy, MinimalBy,ReverseSortBy,SortBy,TakeLargestBy,TakeSmallestBy, Function |

For the *By-type function, ResourceFunction["MonitorProgress"] will monitor only the progress of applying the function to the individual items. Anything else, e.g. the actual sorting, is not included.

For the functions listed in "operator forms", ResourceFunction["MonitorProgress"] will return an operator if the expression was an operator form.

For some types of expressions, ResourceFunction["MonitorProgress"] cannot determine the number of expected steps. In these cases, an "endless" progress bar is used.

ResourceFunction["MonitorProgress"] accepts the following options:

"Resolution" | Automatic | resolution with which to update the progress bar |

"Timing" | True | whether to include timing information |

"Label" | None | label for the progress indicator |

"CurrentDisplayFunction" | Full | how to display the current item of the computation |

Parallelization | Automatic | how to handle parallelization of the evaluation |

The option "Resolution" can be set to the following:

steps | update the progress bar after the specified number of steps |

Scaled[frac] | update after a fraction of the total number of steps |

Automatic | use an automatically determined setting |

The setting "Resolution"→Automatic is equivalent to Scaled[1/20] when the number of steps is known, and 5 otherwise.

The option "Timing" can be set to the following:

True | include timing information |

False | do not include timing information |

Automatic | only include timing information if the total number of steps is known |

The timing information provided includes time elapsed, time per step, estimated time remaining and estimated total time. The latter two are missing if the number of steps is unknown.

The option "CurrentDisplayFunction" can be set to the following:

Full | show a short version of the current item, with a tooltip of the full item |

Automatic | do not include a tooltip |

None | do not display the current item |

func | apply func before generating the tooltipped short version of the current item |

{func} | same as func, but do not include a tooltip |

{func,toolfunc} | apply toolfunc to generate the content of the tooltip |

{func,Full} | show the full current item in the tooltip |

The function specified by "CurrentDisplayFunction" is effectively applied to the value set by ResourceFunction["MonitorProgress"]["SetCurrent"]/ResourceFunction["MonitorProgress"]["SetCurrentBy"].

ResourceFunction["MonitorProgress"] supports parallelized evaluation, with the progress being properly synchronized between kernels. The exact behavior can be specified using the Parallelization option:

Automatic | only use parallelization if the argument is explicitly parallelized |

False | disable parallelization support |

{opts} | attempt parallelization using Parallelize with the specified options |

Full | attempt parallelization using Parallelize; equivalent to {} |

If Parallelization→Automatic is specified, any parallelization options specified in *expr* are taken into account.

If ResourceFunction["MonitorProgress"] cannot automatically determine how to inject tracking into the evaluated expression, the necessary information can be provided manually. Apart from supplying the number of steps expected, it is also necessary to indicate when each step of the computation is finished. It is also possible to set a label for the currently processed item/step. The following functions can be used for this purpose:

ResourceFunction["MonitorProgress"]["Step"][args] | indicates that one step is done; returns args |

ResourceFunction["MonitorProgress"]["SetCurrent"][label] | set the label of the current step to label; returns label |

ResourceFunction["MonitorProgress"]["SetCurrentBy"][func][expr] | set the label of the current step to func[expr];
returns expr |

All three functions for manual monitoring return their arguments to allow for simple chaining.

If the number of steps is unknown, Indeterminate can be specified for *steps* instead. This causes an "endless" progress bar to be used.

ResourceFunction["MonitorProgress"] supports advanced evaluation control and will work nicely with Unevaluated arguments and functions with HoldAll and related attributes.

ResourceFunction["MonitorProgress"] will accurately predict the number of steps needed for mapping operations on SparseArray expressions.

Even though ResourceFunction["MonitorProgress"] is designed to incur minimal overhead, the performance impact might not be negligible if the individual steps are very fast. Furthermore, ResourceFunction["MonitorProgress"] might prevent automatic compilation performed by some functions.

Monitor the progress of a Map operation:

In[1]:= |

Out[1]= |

Out[1]= |

Monitor the progress of a 2D Table generation:

In[2]:= |

Out[2]= |

Out[2]= |

In some cases, the total number of steps is not known:

In[3]:= |

Out[3]= |

Out[5]= |

Create an operator for later use:

In[6]:= |

Out[6]= |

Apply the operator to a list:

In[7]:= |

Out[7]= |

Out[7]= |

Indicate steps manually for cases where they cannot be automatically inferred:

In[8]:= |

Out[8]= |

MonitorProgress supports advanced evaluation control and does not evaluate any arguments to functions with Hold attributes:

In[9]:= |

Out[9]= |

Out[11]= |

Mapping operations on SparseArray expressions are also supported, and the correct number of steps is determined:

In[12]:= |

Out[12]= |

Out[12]= |

MonitorProgress will attempt to construct tracked operators out of Query expressions:

In[13]:= |

Out[13]= |

In[14]:= |

Out[14]= |

Out[14]= |

Monitor a Dataset query without having to wrap the whole argument explicitly in MonitorProgress@Query[…]:

In[15]:= |

Out[15]= |

Out[15]= |

With the default setting "CurrentDisplayFunction"→Full, a short version of the current item is displayed with a tooltip of the full item:

In[16]:= |

Out[16]= |

Do not include the tooltip:

In[17]:= |

Out[17]= |

Do not display the current item at all (this can improve performance for complex items):

In[18]:= |

Out[18]= |

Use a custom function to generate the expression to display:

In[19]:= |

Out[19]= |

Use a custom function to generate the expression to display, but do not include a tooltip:

In[20]:= |

Out[20]= |

Use a different function to generate the tooltip content:

In[21]:= |

Out[21]= |

Show the full, unmodified current item in the tooltip:

In[22]:= |

Out[22]= |

With the default setting "Label"→None, no label is included:

In[23]:= |

Out[23]= |

Out[23]= |

Set a custom label for the progress indicator:

In[24]:= |

Out[24]= |

Out[25]= |

With the default setting Parallelization→Automatic, parallelization support is only enabled for explicitly parallelized computations:

In[26]:= |

Out[26]= |

In[27]:= |

Out[27]= |

Attempt explicit parallelization using the specified options:

In[28]:= |

Out[28]= |

This is equivalent to explicitly wrapping the argument in Parallelize[…,opts]:

In[29]:= |

Out[29]= |

Attempt parallelization using the default settings:

In[30]:= |

Out[30]= |

With the default setting "Resolution"→Automatic, the progress bar is updated 20 times:

In[31]:= |

Out[31]= |

Update the progress bar only five times:

In[32]:= |

Out[32]= |

Update the progress bar after every single step:

In[33]:= |

Out[33]= |

MonitorProgress computes the total time needed based on the average of all previous steps, which leads to inaccurate predictions for highly variable step durations:

In[38]:= |

Out[38]= |

Timing information can be disabled in these cases:

In[39]:= |

Out[39]= |

Although MonitorProgress will return an operator for arbitrary Function expressions, the resulting expression must still be supported by MonitorProgress:

In[40]:= |

Out[40]= |

In[41]:= |

Out[41]= |

MonitorProgress does not support all types of Query expressions:

In[42]:= |

Out[42]= |

MonitorProgress tracks queries by effectively applying Normal to them. This means that certain special behaviors of Query may not be faithfully reproduced:

In[43]:= |

Out[16]= |

In[44]:= |

Out[44]= |

In[45]:= |

Out[45]= |

Although the overhead of MonitorProgress is low, it might not be negligible in all cases:

In[46]:= |

In[47]:= |

Out[47]= |

Very extreme cases of slowdown occur when the untracked operation could be automatically compiled, since the tracking prevents this auto-compilation:

In[48]:= |

Out[48]= |

In[49]:= |

Out[49]= |

Parallelization might fail if the underlying operation cannot be parallelized:

In[50]:= |

Out[50]= |

- 2.2.1 – 15 March 2023
- 2.2.0 – 16 August 2022
- 2.1.2 – 09 August 2022
- 2.1.1 – 18 July 2022
- 2.1.0 – 05 July 2022
- 2.0.0 – 01 September 2020
- 1.0.0 – 18 December 2019

This work is licensed under a Creative Commons Attribution 4.0 International License