Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

 GenerateJUnitReport

Source Notebook

Convert a TestReport to an exportable XML document

Contributed by: Franco Peña Campos

ResourceFunction["GenerateJUnitReport"][report]

generates a JUnit report XML document using report.

Details and Options

JUnit reports can be imported by other testing tools.
Originally JUnit was exclusive to exporting Java JUnit test reports. However, it is now used by other languages and frameworks like Selenium or Playwright.
JUnit report defines two different types of issues when running tests: "errors" and "failures". A test can only "fail" once, but it can have multiple "errors". TestObject doesn't define the concept of "error" so this is not used when generating JUnit reports.
In the JUnit report, "hostname" field will always be "Default". "hostname" can be understood like the environment where the tests are being run and this is not something that we can obtain from a TestReportObject.
By default, test suites will be formed by grouping tests by their "TestFileName" (see the TestObject documentation details). Tests belonging to the same file name will be considered from the same test suite. If the tests don't define a "TestFileName" field, then a single test suite called "None" will group all the tests in the report.
TestObject "AbsoluteTimeUsed" is used to define the amount of time each test took. Each test suite uses the sum of their tests times to define the total time used. If "AbsoluteTimeUsed" is not available for a specific test, zero is used by default.
Each test failure contains "CDATA" xml block with the expected and actual behavior of the test.
Ideally, TestID should only contain alphanumeric characters and basic punctuation characters ("-_.,"). To avoid XML compatibility issues, all special/uncommon characters are escaped in the generated report.
ResourceFunction["GenerateJUnitReport"] takes the following options:
"TestSuiteNameFunction"Automaticfunction that returns the name for each test suite
"Verbose"Truewhether to include the actual and the expected behavior in the report
The "TestSuitNameFunction" function is called with 2 arguments: the file name and a list of TestObject expression

Examples

Basic Examples (2) 

Generate a JUnit report from a TestReportObject:

In[1]:=
junitReport = ResourceFunction["GenerateJUnitReport"][ TestReportObject[<|"Title" -> Automatic, "Aborted" -> False, "TestResults" -> <|2478462031575520723 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.903022614021387*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "cdedb05a-5509-4fd8-b1ba-980a2b2b074c", "TestID" -> "Range", "TestFileName" -> "", "EvaluationID" -> "947de1dd-6d45-4bc5-954b-b2370ec87a9d", "Input" -> HoldForm[ Range[10]], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000048, "CPUTimeUsed" -> 0.000046999999995023245`, "MemoryUsed" -> 2824, "ExpectedOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "Outcome" -> "Success"|>], 4106657113249137150 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.903022614021725*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "35c6101d-3e8d-452f-8008-fcbdf7efa940", "TestID" -> "Sum", "TestFileName" -> "", "EvaluationID" -> "a535b21c-eda0-4be1-8f09-c2183572783d", "Input" -> HoldForm[2 + 2], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[4], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000035, "CPUTimeUsed" -> 0.00003399999999942338, "MemoryUsed" -> 2584, "ExpectedOutput" -> HoldForm[4], "Outcome" -> "Success"|>], 5447270481595823336 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.903022614022017*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "f8be08a2-3e2d-4d2e-99fa-fdd47505d86e", "TestID" -> "Fraction", "TestFileName" -> "", "EvaluationID" -> "cc47f654-539b-4679-a0cf-77ff0a50ae47", "Input" -> HoldForm[1/4.], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[0.25], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000037, "CPUTimeUsed" -> 0.00003799999999642978, "MemoryUsed" -> 2608, "ExpectedOutput" -> HoldForm[0.25], "Outcome" -> "Success"|>]|>, "FailureResults" -> <||>, "TestsNotEvaluatedKeys" -> {}, "TestsFailedWrongResultsKeys" -> {}, "TestsFailedWithMessagesKeys" -> {}, "TestsFailedWithErrorsKeys" -> {}, "TestsSucceededKeys" -> {2478462031575520723, 4106657113249137150, 5447270481595823336}|>]]
Out[1]=

Export the XML as a string:

In[2]:=
ExportString[junitReport, "XML"]
Out[2]=

Scope (2) 

Generate a JUnit report from a TestReportObject:

In[3]:=
junitReport = ResourceFunction["GenerateJUnitReport"][ TestReportObject[<|"Title" -> Automatic, "Aborted" -> False, "TestResults" -> <|1040801227156669071 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.900767331468316*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "b08f6807-c155-4c30-b6b4-dd7bb8c6ac0b", "TestID" -> "my test", "TestFileName" -> "", "EvaluationID" -> "b9059fc7-e167-4fd3-8442-603e9bcc27d4", "Input" -> HoldForm[ Range[10]], "ExpectedOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000051, "CPUTimeUsed" -> 0.00005099999999913507, "MemoryUsed" -> 2456, "Outcome" -> "Success"|>], 510008052212128275 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.9007673314684944`*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "92413500-1e7a-48b5-902f-07d2af26d5e0", "TestID" -> None, "TestFileName" -> "", "EvaluationID" -> "977e799c-f629-4b6d-b64c-36d56e14fa25", "Input" -> HoldForm[ RandomInteger[10]], "ExpectedOutput" -> HoldForm[ Blank[Integer]], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[5], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000632, "CPUTimeUsed" -> 0.00016800000000039006`, "MemoryUsed" -> 5360, "Outcome" -> "Failure"|>], 2756168186189522548 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.900767331468634*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "b955ac1e-f63d-419c-a7df-aa467d80611d", "TestID" -> None, "TestFileName" -> "", "EvaluationID" -> "32ab4231-d7f2-4fb1-bdeb-52c8c6b75272", "Input" -> HoldForm[1/0], "ExpectedOutput" -> HoldForm[ DirectedInfinity[]], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[ DirectedInfinity[]], "ActualMessages" -> { HoldForm[ Message[ MessageName[Power, "infy"], HoldForm[0^(-1)]]]}, "AbsoluteTimeUsed" -> 0.023333, "CPUTimeUsed" -> 0.02332399999999879, "MemoryUsed" -> 11248, "Outcome" -> "MessagesFailure"|>]|>, "FailureResults" -> <||>, "TestsNotEvaluatedKeys" -> {}, "TestsFailedWithErrorsKeys" -> {}, "TestsSucceededKeys" -> {1040801227156669071}, "TestsFailedWrongResultsKeys" -> {510008052212128275}, "TestsFailedWithMessagesKeys" -> {2756168186189522548}|>]];

Export to an XML document for use in other applications:

In[4]:=
Export["junit-report.xml", junitReport]
Out[4]=

Options (3) 

TestSuiteNameFunction (1) 

Customize test suites names when generating the report:

In[5]:=
customTestSuiteName[fileName_, testCases_] := "CustomPrefix-" <> FileBaseName[fileName]
In[6]:=
junitReport = ResourceFunction["GenerateJUnitReport"][ TestReportObject[<|"Title" -> Automatic, "Aborted" -> False, "TestResults" -> <|5653498412255593124 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.9018311620338497`*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "0fb9ccaa-6118-4e74-a9a9-da822c71cda0", "TestID" -> "Abstract sum", "TestFileName" -> "/tests/MyImportantTest.m", "EvaluationID" -> "1e6109c9-28f6-4590-aea1-960fc828173b", "Input" -> HoldForm[1 + 1], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[2], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000053, "CPUTimeUsed" -> 0.000053999999977349944`, "MemoryUsed" -> 2584, "ExpectedOutput" -> HoldForm[3], "Outcome" -> "Failure"|>]|>, "FailureResults" -> <||>, "TestsNotEvaluatedKeys" -> {}, "TestsSucceededKeys" -> {}, "TestsFailedWithMessagesKeys" -> {}, "TestsFailedWithErrorsKeys" -> {}, "TestsFailedWrongResultsKeys" -> {5653498412255593124}|>], "TestSuiteNameFunction" -> customTestSuiteName];
In[7]:=
ExportString[junitReport, "XML"]
Out[7]=

Verbose (2) 

Disable verbose to avoid exporting too much data in the report. For example, this report contains more than 1MB of data:

In[8]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/dfc697c2-325d-42a2-9089-46b0f09e9636"]
Out[8]=

By disabling "Verbose" we can get a more compact report with its key components:

In[9]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/63857acb-f2a8-4e8e-9e30-4989f5fe1ea9"]
Out[9]=

Properties and Relations (1) 

The resulting XMLObject always contains some default values which can not be determined from a TestResultObject. Note the "name", "hostname" and "errors" values:

In[10]:=
junitReport = ResourceFunction["GenerateJUnitReport"][ TestReportObject[<|"Title" -> Automatic, "Aborted" -> False, "TestResults" -> <|1040801227156669071 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.900767331468316*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "b08f6807-c155-4c30-b6b4-dd7bb8c6ac0b", "TestID" -> "my test", "TestFileName" -> "", "EvaluationID" -> "b9059fc7-e167-4fd3-8442-603e9bcc27d4", "Input" -> HoldForm[ Range[10]], "ExpectedOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000051, "CPUTimeUsed" -> 0.00005099999999913507, "MemoryUsed" -> 2456, "Outcome" -> "Success"|>], 510008052212128275 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.9007673314684944`*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "92413500-1e7a-48b5-902f-07d2af26d5e0", "TestID" -> None, "TestFileName" -> "", "EvaluationID" -> "977e799c-f629-4b6d-b64c-36d56e14fa25", "Input" -> HoldForm[ RandomInteger[10]], "ExpectedOutput" -> HoldForm[ Blank[Integer]], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[5], "ActualMessages" -> {}, "AbsoluteTimeUsed" -> 0.000632, "CPUTimeUsed" -> 0.00016800000000039006`, "MemoryUsed" -> 5360, "Outcome" -> "Failure"|>], 2756168186189522548 -> TestObject[<|"MetaInformation" -> None, "AbsoluteTime" -> 3.900767331468634*^9, "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, "MemoryConstraint" -> DirectedInfinity[1], "TimeConstraint" -> DirectedInfinity[1], "CreationID" -> "b955ac1e-f63d-419c-a7df-aa467d80611d", "TestID" -> None, "TestFileName" -> "", "EvaluationID" -> "32ab4231-d7f2-4fb1-bdeb-52c8c6b75272", "Input" -> HoldForm[1/0], "ExpectedOutput" -> HoldForm[ DirectedInfinity[]], "ExpectedMessages" -> HoldForm[{}], "ActualOutput" -> HoldForm[ DirectedInfinity[]], "ActualMessages" -> { HoldForm[ Message[ MessageName[Power, "infy"], HoldForm[0^(-1)]]]}, "AbsoluteTimeUsed" -> 0.023333, "CPUTimeUsed" -> 0.02332399999999879, "MemoryUsed" -> 11248, "Outcome" -> "MessagesFailure"|>]|>, "FailureResults" -> <||>, "TestsNotEvaluatedKeys" -> {}, "TestsFailedWithErrorsKeys" -> {}, "TestsSucceededKeys" -> {1040801227156669071}, "TestsFailedWrongResultsKeys" -> {510008052212128275}, "TestsFailedWithMessagesKeys" -> {2756168186189522548}|>]];
In[11]:=
testSuiteMetadata = junitReport[[2]][[3]][[1]][[2]]
Out[11]=

Publisher

Franco Pena-Campos

Requirements

Wolfram Language 13.0 (December 2021) or above

Version History

  • 2.0.0 – 06 September 2024
  • 1.1.1 – 08 November 2023
  • 1.1.0 – 13 September 2023
  • 1.0.0 – 01 September 2023

Related Resources

Author Notes

This tool is intended to improve TestReportObject by allowing it to be exported to other test frameworks.

License Information