matlab.unittest.plugins.StopOnFailuresPlugin class

Package: matlab.unittest.plugins

Plugin to debug test failures

expand all in page

Description

The StopOnFailuresPlugin class provides a plugin to help debug test failures. Adding StopOnFailuresPlugin to the test runner pauses execution of a test if it encounters a qualification failure or uncaught error and puts MATLAB® into debug mode.

If StopOnFailuresPlugin encounters a qualification failure or uncaught error in a test, you can use MATLAB debugging commands, such as dbstep, dbcont, and dbquit, to investigate the cause of the test failure. In the case of an uncaught error in a test, you cannot use dbup to shift context to the source of the test failure, because the error disrupts the stack.

Construction

matlab.unittest.plugins.StopOnFailuresPlugin creates a plugin to debug test failures.

matlab.unittest.plugins.StopOnFailuresPlugin('IncludingAssumptionFailures',tf) indicates whether to react to assumption failures. By default, StopOnFailuresPlugin reacts to only uncaught errors and verification, assertion, and fatal assertion qualification errors. However, when 'IncludingAssumptionFailures' is set to true, the plugin also reacts to assumption failures.

Input Arguments

expand all

Indicator to react to assumption failures, specified as logical false or true. When this value is true, the test runner reacts to assumption failures. When the value is false, the plugin ignores assumption failures.

Properties

IncludeAssumptionFailures

When this property value is true, the instance reacts to assumption failures. When the value is false, the instance ignores assumption failures. The IncludeAssumptionFailures property is false by default. To specify the property as true, use the IncludingAssumptionFailures input when you construct the instance.

Copy Semantics

Handle. To learn how handle classes affect copy operations, see Copying Objects.

Examples

collapse all

In your working folder, create the file ExampleTest.m containing the following test class.

classdef ExampleTest < matlab.unittest.TestCase
    methods(Test)
        function testOne(testCase)  % Test fails
            act = 3.1416;
            exp = pi;
            testCase.verifyEqual(act, exp)
        end
        function testTwo(testCase)  % Test does not complete
            testCase.assumeEqual(5, 4)
        end
    end
end

At the command prompt, create a test suite from the ExampleTest class and a test runner.

import matlab.unittest.TestRunner
import matlab.unittest.TestSuite
import matlab.unittest.plugins.StopOnFailuresPlugin

suite = TestSuite.fromClass(?ExampleTest);
runner = TestRunner.withTextOutput;

Run the tests.

result = runner.run(suite);
Running ExampleTest

================================================================================
Verification failed in ExampleTest/testOne.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The values are not equal using "isequaln".
    --> Failure table:
                Actual        Expected               Error               RelativeError    
                ______    ________________    ____________________    ____________________
            
                3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06
    
    Actual double:
           3.141600000000000
    Expected double:
           3.141592653589793

    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================
.
================================================================================
ExampleTest/testTwo was filtered.
    Details
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name                 Failed  Incomplete  Reason(s)
    ==================================================================
     ExampleTest/testOne    X                 Failed by verification.
    ------------------------------------------------------------------
     ExampleTest/testTwo              X       Filtered by assumption.

As a result of the qualifications in the test class, the first test fails, and the second test does not complete.

Add the StopOnFailuresPlugin to the runner and run the tests.

runner.addPlugin(StopOnFailuresPlugin)
result = runner.run(suite);
Running ExampleTest
Test execution paused due to failure. Either click here or execute DBUP to shift context to its source: line 6 of "C:\work\ExampleTest.m".

During the test execution, when the failure occurs, MATLAB enters debug mode.

Click on the hyperlinked word 'here' to shift debug context to your work source. If necessary, make the command window your current window.

In workspace belonging to ExampleTest>ExampleTest.testOne at 6

Examine the variables in the workspace.

whos
  Name          Size            Bytes  Class           Attributes

  act           1x1                 8  double                    
  exp           1x1                 8  double                    
  testCase      1x1               112  ExampleTest

Now, you can investigate the cause of the test failure.

For example, see if the test passes when you specify a relative tolerance of 100*eps.

testCase.verifyEqual(act,exp,'RelTol',100*eps)
Verification failed in ExampleTest/testOne.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The values are not equal using "isequaln".
    --> The error was not within relative tolerance.
    --> Failure table:
                Actual        Expected               Error               RelativeError         RelativeTolerance  
                ______    ________________    ____________________    ____________________    ____________________
            
                3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06    2.22044604925031e-14
    
    Actual double:
           3.141600000000000
    Expected double:
           3.141592653589793

    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================

The test fails even with the specified tolerance.

Exit out of debug mode.

dbquit
================================================================================
Verification failed in ExampleTest/testOne.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The values are not equal using "isequaln".
    --> Failure table:
                Actual        Expected               Error               RelativeError    
                ______    ________________    ____________________    ____________________
            
                3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06
    
    Actual double:
           3.141600000000000
    Expected double:
           3.141592653589793

    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================
.
================================================================================
ExampleTest/testTwo was filtered.
    Details
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name                 Failed  Incomplete  Reason(s)
    ==================================================================
     ExampleTest/testOne    X                 Failed by verification.
    ------------------------------------------------------------------
     ExampleTest/testTwo              X       Filtered by assumption.

To enter debug mode for tests that fail by assumption, such as testTwo in the ExampleTest class, include 'IncludingAssumptionFailures' option for the plugin.

runner = TestRunner.withTextOutput;
runner.addPlugin(StopOnFailuresPlugin(...
    'IncludingAssumptionFailures', true))

If you run the test runner, you enter debug mode for both testOne and testTwo.

See Also

| | | |

Introduced in R2013b

MATLAB Documentation

Support