Using the Testing Framework

Whether you are writing a single function or an entire application, Mathematica provides a framework in which to write and run tests of your code.

Getting Started

In the Mathematica menus, click on File ▶ New ▶ Testing Notebook. It will bring up a testing notebook. (The notebook depicted here has also been saved using File ▶ Save As... with name MyTests.)

Click the New button to insert a test into the notebook.

In the "Input" cell, type or paste the input for your test. In the "Expected Output" cell, type or paste what the output should be.

If messages are expected to be generated upon running the tests, add them using the Add Messages button. You can also add options to the individual tests such as (RefLink[MemoryConstraint, paclet:ref/MemoryConstraint]), (RefLink[SameTest, paclet:ref/SameTest]), (RefLink[TestID, paclet:ref/TestID]), and (RefLink[TimeConstraint, paclet:ref/TimeConstraint]) using the drop-down to the right of the Add Options button.

Running Tests with the Notebook Interface

This section demonstrates running all tests in a given notebook.

Clicking the Run button will evaluate all tests in the testing notebook. The test results are inserted into the individual tests as they are evaluated, and upon completion of the test run, a summary of the test report is inserted at the top of the notebook. The following notebook displays the four possible test outcomes: Success, Failure, Message Failure, and Error. The Clear Test Results button may be used to clear test results from the notebook.

The test report summary details the total number of tests run, as well as the number of successes, errors, failures due to wrong results, and failures due to messages. Additionally, a summary bar depicts the test results and has two modes for ordering the test results: by status and by sequence. When the total number of tests in a testing notebook is 50 or less, every individual tab in the summary bar links to its appropriate test. In the lower-right corner, there are two buttons that find the previous and next failed test relative to the cursor position.

Working with Test Results

Following is a success. If you wished to modify this particular test and see the outcome, you would then use the Rerun button.

Following is a failure due to a wrong result. This happens when the Actual Output is different from the Expected Output. If the Actual Output is the correct result, the Expected Output can be changed to it by clicking on the Replace Output button. Subsequently, the test can be rerun by using the Rerun button.

Following is a Message Failure. Here there is an Actual Message. The test fails since there is no Expected Messages cell with that message. The Replace Message List button may be used to insert an Expected Messages cell with content as in the Actual Messages cell. The test may then be rerun using the Rerun button.

Following is a test Error. Fix the right-hand side of the option in the Test Options cell and use the Rerun button.

Generating .wlt Files

Click More ▶ Save As .wlt to save the tests in the notebook to a .wlt file. A file save chooser will be generated, allowing you to choose the file name and save it to a directory.

Here the file name was chosen to be MyTests2 and it was saved to the directory C:\\Users\\xyz\\Desktop. This makes a report for that file.

tr=TestReport["C:\\Users\\xyz\\Desktop\\MyTests2.wlt"]

Here are the properties of the report.

tr["Properties"]

Here are the test result rules.

tr["TestResultRules"]

This shows the expected output and actual output for the first failed test. For this test file there is just one failed test.

ft=tr["TestsFailedWrongResults"][[1]]

ft/@{"ExpectedOutput","ActualOutput"}

Assigning TestIDs

Click More ▶ Assign TestIDs to assign TestIDs to all tests in the notebook.

Here is a test cell group with a TestID assigned.

Notebook Conversion

You might want to convert a notebook of inputs, messages, and outputs into a test notebook. With that notebook being the input notebook, click on Window ▶ Toolbar ▶ Testing. A docked cell will then be added to the top of that notebook. Then click on the button Convert to Testing Notebook.

Directly Running Tests in a Notebook

Single tests may be directly run using (RefLink[VerificationTest, paclet:ref/VerificationTest]).

VerificationTest[Solve[x^2+a x+1==0,x],{{x->(1/2) (-a-(Sqrt[-4+(a^2)]))},{x->(1/2) (-a+(Sqrt[-4+(a^2)]))}}]

Multiple tests may be directly run using (RefLink[TestReport, paclet:ref/TestReport]).

TestReport[{VerificationTest[Sin[E]>Cos[E]],VerificationTest[MatrixQ[{{1,2,3},{4,5,6,7}}]],VerificationTest[1/0,ComplexInfinity]}]