Wolfram Computation Meets Knowledge

MonitoredTestReport

Contributed by: Richard Hennigan (Wolfram Research)

Generate a TestReportObject with dynamic progress

ResourceFunction["MonitoredTestReport"][{test1,test2,}]

gives a report of the results of the verification tests testi while displaying test progress.

ResourceFunction["MonitoredTestReport"]["file"]

runs tests from a file, returning an analysis of the completed test run.

Details and Options

MonitoredTestReport returns a TestReportObject.
MonitoredTestReport has the following options, which are applied to each VerificationTest it runs:
MemoryConstraint Infinity memory (in bytes) each test is allowed to use
SameTest SameQ function to compare actual and expected outputs
TimeConstraint Infinity time (in seconds) each test is allowed to use
Additionally, MonitoredTestReport has the option "Title", which is added to the metadata of the TestReportObject.
"Title" Automatic sets the title of the TestReportObject
MonitoredTestReport is functionally similar to using TestReport, but it provides information about testing progress. Additionally, it allows testing to be stopped early to return results that have been completed so far.

Examples

Basic Examples

View progress information while running a test report:

In[1]:=
ResourceFunction["MonitoredTestReport"][{
  VerificationTest[Sin[E] > Cos[E]],
  VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]],
  VerificationTest[1/0, ComplexInfinity]
  }]
Out[2]=

Run tests from a notebook:

In[3]:=
report = ResourceFunction["MonitoredTestReport"][
  FileNameJoin[{"ExampleData", "ExampleTestFile.nb"}]]
Out[3]=

View properties of the TestReportObject:

In[4]:=
report["Properties"]
Out[4]=

Get an Association of individual test results in the form of TestResultObject:

In[5]:=
report["TestResults"]
Out[5]=

Extract information on tests that failed and present in a formatted TabView:

In[6]:=
TabView@Normal@Map[TabView@*Replace[{} -> {"" -> ""}]@*Normal@*Map[
ResourceFunction[
ResourceObject[
Association[
       "Name" -> "ReadableInputForm", "UUID" -> "2853fd6d-28bf-4480-87ae-720dd1c23276", "ResourceType" -> "Function", "Version" -> "1.0.0", "Description" -> "Display an expression in a nicely formatted \
input form", "RepositoryLocation" -> URL[
         "https://www.wolframcloud.com/objects/resourcesystem/api/1.\
0"], "WolframLanguageVersionRequired" -> "10.0", "SymbolName" -> "FunctionRepository`$\
441d762676974f19ae3e73efb3b3b435`ReadableInputForm", "FunctionLocation" -> CloudObject[
         "https://www.wolframcloud.com/objects/dab710ac-fead-4879-\
9298-42d246db8772"], "ShortName" -> "ReadableInputForm"], ResourceSystemBase -> "https://www.wolframcloud.com/objects/\
resourcesystem/api/1.0"]]], report["TestsFailed"]]
Out[6]=

More Examples

Scope

Retrieve the ResourceFunction:

In[7]:=
ResourceFunction["MonitoredTestReport"]
Out[7]=

Retrieve the ResourceObject:

In[8]:=
ResourceObject["MonitoredTestReport"]
Out[8]=

Verification tests can also be wrapped in HoldComplete, which is useful for constructing lists of tests without evaluation:

In[9]:=
ResourceFunction["MonitoredTestReport"][{
  HoldComplete[VerificationTest[Sin[E] > Cos[E]]],
  HoldComplete[VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]]],
  HoldComplete[VerificationTest[1/0, ComplexInfinity]]
  }]
Out[9]=

Generalizations & Extensions

Pressing the stop button during testing will return a TestReportObject containing the tests that were completed so far:

In[10]:=
tests = ConstantArray[HoldComplete@VerificationTest[Pause[1], 1], 500];
ResourceFunction["MonitoredTestReport"][tests]
Out[11]=

Applications

Here's an API that we would like to test:

In[12]:=
api = CloudDeploy[APIFunction[{"x" -> "Integer"}, #! &], "testAPI"]
Out[12]=

Here's a file containing (incorrect) tests for it:

In[13]:=
testFile = FileNameJoin@{$TemporaryDirectory, "apiTests.wlt"};
Scan[PutAppend[
    Unevaluated[
     VerificationTest[
      URLExecute[CloudObject["testingAPI"], {"x" -> #}], #!]], testFile] &, Range[1000]];
FilePrint[testFile, 5]

Use the progress panel to quickly spot when something is wrong in order to stop early:

In[14]:=
ResourceFunction["MonitoredTestReport"][testFile]
Out[14]=

Resource History