Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

 MonitoredTestReport

Source Notebook

Generate a TestReportObject with dynamic progress

Contributed by: Richard Hennigan (Wolfram Research)

ResourceFunction["MonitoredTestReport"][{test1,test2,…}]

gives a report of the results of the verification tests testi while displaying test progress.

ResourceFunction["MonitoredTestReport"]["file"]

runs tests from a file, returning an analysis of the completed test run.

Details and Options

MonitoredTestReport returns a TestReportObject.
MonitoredTestReport has the following options, which are applied to each VerificationTest it runs:
MemoryConstraintInfinitymemory (in bytes) each test is allowed to use
SameTestSameQfunction to compare actual and expected outputs
TimeConstraintInfinitytime (in seconds) each test is allowed to use
Additionally, MonitoredTestReport has the option "Title", which is added to the metadata of the TestReportObject:
"Title"Automaticsets the title of the TestReportObject
MonitoredTestReport is functionally similar to using TestReport, but it provides information about testing progress. Additionally, it allows testing to be stopped early to return results that have been completed so far.

Examples

Basic Examples (5) 

View progress information while running a test report:

In[1]:=
ResourceFunction["MonitoredTestReport"][{ VerificationTest[Sin[E] > Cos[E]], VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]], VerificationTest[1/0, ComplexInfinity] }]
Out[1]=

Run tests from a notebook:

In[2]:=
report = ResourceFunction["MonitoredTestReport"][ FileNameJoin[{"ExampleData", "ExampleTestFile.nb"}]]
Out[2]=

View properties of the TestReportObject:

In[3]:=
report["Properties"]
Out[3]=

Get an Association of individual test results in the form of TestResultObject:

In[4]:=
report["TestResults"]
Out[4]=

Extract information on tests that failed and present in a formatted TabView:

In[5]:=
TabView@Normal@Map[TabView@*Replace[{} -> {"" -> ""}]@*Normal@*Map[ ResourceFunction[ ResourceObject[<|{"Name" -> "ReadableInputForm", "UUID" -> "2853fd6d-28bf-4480-87ae-720dd1c23276", "ResourceType" -> "Function", "Version" -> "1.0.0", "Description" -> "Display an expression in a nicely formatted input form", "RepositoryLocation" -> URL[ "https://www.wolframcloud.com/objects/resourcesystem/api/1.0"], "WolframLanguageVersionRequired" -> "10.0", "SymbolName" -> "FunctionRepository`$441d762676974f19ae3e73efb3b3b435`ReadableInputForm", "FunctionLocation" -> CloudObject[ "https://www.wolframcloud.com/objects/dab710ac-fead-4879-9298-42d246db8772"], "ShortName" -> "ReadableInputForm"}|>, ResourceSystemBase -> "https://www.wolframcloud.com/objects/resourcesystem/api/1.0"]]], report["TestsFailed"]]
Out[5]=

Scope (2) 

Retrieve the ResourceFunction:

In[6]:=
ResourceFunction["MonitoredTestReport"]
Out[6]=

Retrieve the ResourceObject:

In[7]:=
ResourceObject["MonitoredTestReport"]
Out[7]=

Verification tests can also be wrapped in HoldComplete, which is useful for constructing lists of tests without evaluation:

In[8]:=
ResourceFunction["MonitoredTestReport"][{ HoldComplete[VerificationTest[Sin[E] > Cos[E]]], HoldComplete[VerificationTest[MatrixQ[{{1, 2, 3}, {4, 5, 6, 7}}]]], HoldComplete[VerificationTest[1/0, ComplexInfinity]] }]
Out[8]=

Generalizations and Extensions (1) 

Pressing the stop button during testing will return a TestReportObject containing the tests that were completed so far:

In[9]:=
tests = ConstantArray[HoldComplete@VerificationTest[Pause[1], 1], 500]; ResourceFunction["MonitoredTestReport"][tests]
Out[10]=

Applications (3) 

Here's an API that we would like to test:

In[11]:=
api = CloudDeploy[APIFunction[{"x" -> "Integer"}, #! &], "testAPI"]
Out[11]=

Here's a file containing (incorrect) tests for it:

In[12]:=
testFile = FileNameJoin@{$TemporaryDirectory, "apiTests.wlt"}; Scan[PutAppend[ Unevaluated[ VerificationTest[ URLExecute[CloudObject["testingAPI"], {"x" -> #}], #!]], testFile] &, Range[1000]]; FilePrint[testFile, 5]

Use the progress panel to quickly spot when something is wrong in order to stop early:

In[13]:=
ResourceFunction["MonitoredTestReport"][testFile]
Out[13]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Version History

  • 1.0.0 – 28 August 2018

Related Resources

License Information