Wolfram Research

Function Repository Resource:

MonitoredTestReport

Source Notebook

Generate a TestReportObject with dynamic progress

Contributed by: Richard Hennigan (Wolfram Research)

ResourceFunction["MonitoredTestReport"][{test1,test2,}]

gives a report of the results of the verification tests testi while displaying test progress.

ResourceFunction["MonitoredTestReport"]["file"]

runs tests from a file, returning an analysis of the completed test run.

Details and Options

MonitoredTestReport returns a TestReportObject.
MonitoredTestReport has the following options, which are applied to each VerificationTest it runs:
MemoryConstraint Infinity memory (in bytes) each test is allowed to use
SameTest SameQ function to compare actual and expected outputs
TimeConstraint Infinity time (in seconds) each test is allowed to use
Additionally, MonitoredTestReport has the option "Title", which is added to the metadata of the TestReportObject:
"Title" Automatic sets the title of the TestReportObject
MonitoredTestReport is functionally similar to using TestReport, but it provides information about testing progress. Additionally, it allows testing to be stopped early to return results that have been completed so far.

Examples

Basic Examples

View progress information while running a test report:

In[1]:=
ResourceFunction["MonitoredTestReport"][{
  VerificationTest[Sin[E] > Cos[E]],
  VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]],
  VerificationTest[1/0, ComplexInfinity]
  }]
Out[1]=

Run tests from a notebook:

In[2]:=
report = ResourceFunction["MonitoredTestReport"][
  FileNameJoin[{"ExampleData", "ExampleTestFile.nb"}]]
Out[2]=

View properties of the TestReportObject:

In[3]:=
report["Properties"]
Out[3]=

Get an Association of individual test results in the form of TestResultObject:

In[4]:=
report["TestResults"]
Out[4]=

Extract information on tests that failed and present in a formatted TabView:

In[5]:=
TabView@Normal@Map[TabView@*Replace[{} -> {"" -> ""}]@*Normal@*Map[
ResourceFunction[
ResourceObject[
Association[
       "Name" -> "ReadableInputForm", "UUID" -> "2853fd6d-28bf-4480-87ae-720dd1c23276", "ResourceType" -> "Function", "Version" -> "1.0.0", "Description" -> "Display an expression in a nicely formatted \
input form", "RepositoryLocation" -> URL[
         "https://www.wolframcloud.com/objects/resourcesystem/api/1.\
0"], "WolframLanguageVersionRequired" -> "10.0", "SymbolName" -> "FunctionRepository`$\
441d762676974f19ae3e73efb3b3b435`ReadableInputForm", "FunctionLocation" -> CloudObject[
         "https://www.wolframcloud.com/objects/dab710ac-fead-4879-\
9298-42d246db8772"], "ShortName" -> "ReadableInputForm"], ResourceSystemBase -> "https://www.wolframcloud.com/objects/\
resourcesystem/api/1.0"]]], report["TestsFailed"]]
Out[5]=

Scope

Retrieve the ResourceFunction:

In[6]:=
ResourceFunction["MonitoredTestReport"]
Out[6]=

Retrieve the ResourceObject:

In[7]:=
ResourceObject["MonitoredTestReport"]
Out[7]=

Verification tests can also be wrapped in HoldComplete, which is useful for constructing lists of tests without evaluation:

In[8]:=
ResourceFunction["MonitoredTestReport"][{
  HoldComplete[VerificationTest[Sin[E] > Cos[E]]],
  HoldComplete[VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]]],
  HoldComplete[VerificationTest[1/0, ComplexInfinity]]
  }]
Out[8]=

Generalizations and Extensions

Pressing the stop button during testing will return a TestReportObject containing the tests that were completed so far:

In[9]:=
tests = ConstantArray[HoldComplete@VerificationTest[Pause[1], 1], 500];
ResourceFunction["MonitoredTestReport"][tests]
Out[10]=

Applications

Here's an API that we would like to test:

In[11]:=
api = CloudDeploy[APIFunction[{"x" -> "Integer"}, #! &], "testAPI"]
Out[11]=

Here's a file containing (incorrect) tests for it:

In[12]:=
testFile = FileNameJoin@{$TemporaryDirectory, "apiTests.wlt"};
Scan[PutAppend[
    Unevaluated[
     VerificationTest[
      URLExecute[CloudObject["testingAPI"], {"x" -> #}], #!]], testFile] &, Range[1000]];
FilePrint[testFile, 5]

Use the progress panel to quickly spot when something is wrong in order to stop early:

In[13]:=
ResourceFunction["MonitoredTestReport"][testFile]
Out[13]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Resource History

See Also

License Information