Process Test Results with Custom Scripts

Testing your model often requires assessing conditions that ensure a test is valid, in addition to verifying model behavior. MATLAB® Unit Test provides a framework for such assessments. In Simulink® Test™, you can use the test case custom criteria to author specific assessments, and include MATLAB Unit Test qualifications in your script.

Custom criteria apply as post-simulation criteria to the simulation output. If you require run-time verifications, use a verify() statement in a Test Assessment or Test Sequence block. See Assess Model Simulation Using verify Statements.

MATLAB Testing Framework

A custom criteria script is a method of test, which is a matlab.unittest test case object. To enable the function, in the test case Custom Criteria section of the Test Manager, select function customCriteria(test). Inside the function, enter the custom criteria script in the embedded MATLAB editor.

The embedded MATLAB editor lists properties of test. Create test assessments using MATLAB Unit Test qualifications. Custom criteria supports verification and assertion type qualifications. See Table of Verifications, Assertions, and Other Qualifications (MATLAB). Verifications and assertions operate differently when custom criteria are evaluated:

  • Verifications – Other assessments are evaluated when verifications fail. Diagnostics appear in the results. Use verifications for general assessments, such as checking simulation against expected outputs.

    Example: test.verifyEqual(lastValue,0)

  • Assertions – The custom criteria script stops evaluating when an assertion fails. Diagnostics appear in the results. Use assertions for conditions that render the criteria invalid.

    Example: test.assertEqual(lastValue,0).

Define a Custom Criteria Script

This example shows how to create a custom criteria script for an autopilot test case.

  1. Open the test file.

    sltest.testmanager.load('AutopilotTestFile.mldatx')
sltest.testmanager.view

  2. In the Test Browser, select AutopilotTestFile > Basic Design Test Cases > Requirement 1.3 Test. In the test case, expand the Custom Criteria section.

  3. Enable the custom criteria script by selecting function customCriteria(test).

  4. In the embedded MATLAB editor, enter the following script. The script gets the final value of the signals Phi and APEng, and verifies that the final values equal 0.

    % Get the last values
lastPhi = test.sltest_simout.get...
   ('Signals_Req1_3').get('Phi').Values.Data(end);
lastAPEng = test.sltest_simout.get...
   ('Signals_Req1_3').get('APEng').Values.Data(end);

% Verify the last values equal 0
test.verifyEqual(lastPhi,0,...
   ['Final Phi value: ',num2str(lastPhi),'.']);
test.verifyEqual(lastAPEng,false,...
   ['Final APEng value: ',num2str(lastAPEng),'.']);

  5. Run the test case.

  6. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

Reuse Custom Criteria and Debug Using Breakpoints

In addition to authoring criteria scripts in the embedded MATLAB editor, you can author custom criteria in a standalone function, and call the function from the test case. Using a standalone function allows you

  • To reuse the custom criteria in multiple test cases.

  • To set breakpoints in the criteria script for debugging.

  • To investigate the simulation output using the command line.

In this example, you add a breakpoint to a custom criteria script. You run the test case, list the properties of the test object at the command line, and call the custom criteria from the test case.

Call Custom Criteria Script from the Test Case

  1. Navigate to the folder containing the criteria function.

    cd(fullfile(docroot,'toolbox','sltest','examples'))

  2. Open the custom criteria script

    open('sltestCheckFinalRollRefValues.m')

    % This is a custom criteria function for a Simulink Test test case.
% The function gets the last values of Phi and APEng from the 
% Requirements 1.3 test case in the test file AutopilotTestFile.

function sltestCheckFinalRollRefValues(test)

% Get the last values
lastPhi = test.sltest_simout.get...
   ('Signals_Req1_3').get('Phi').Values.Data(end)
lastAPEng = test.sltest_simout.get...
   ('Signals_Req1_3').get('APEng').Values.Data(end)

% Verify the last values equal 0
test.verifyEqual(lastPhi,0,...
   ['Final Phi value: ',num2str(lastPhi),'.']);
test.verifyEqual(lastAPEng,false,...
   ['Final APEng value: ',num2str(lastAPEng),'.']);

  3. Open the test file

    sltest.testmanager.load('AutopilotTestFile.mldatx')
sltest.testmanager.view

  4. In the embedded MATLAB editor under Custom Criteria, enter the function call to the custom criteria:

    sltestCheckFinalRollRefValues(test)

Set Breakpoints and List test Properties

  1. On line 8 of sltestCheckFinalRollRefValues.m, set a breakpoint by clicking the dash to the right of the line number.

  2. In the Test Manager, run the test case.

    The command window displays a debugging prompt.

  3. Enter test at the command prompt to display the properties of the STMCustomCriteria object. The properties contain characteristics and simulation data output of the test case.

    test = 

  STMCustomCriteria with properties:

              TestResult: [1×1 sltest.testmanager.TestCaseResult]
                  sltest_simout: [1×1 Simulink.SimulationOutput]
         sltest_testCase: [1×1 sltest.testmanager.TestCase]
           sltest_bdroot: {'RollReference_Requirement1_3'}
              sltest_sut: {'RollAutopilotMdlRef/Roll Reference'}
        sltest_isharness: 1
    sltest_iterationName: ''

    The property sltest_simout contains the simulation data. To view the data PhiRef, enter

    test.sltest_simout.get('Signals_Req1_3').get('PhiRef')
    ans = 

  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
  struct with fields:

              Name: 'PhiRef'
    PropagatedName: ''
         BlockPath: [1×1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 1
            Values: [1×1 timeseries]

  4. In the MATLAB editor, click Continue to continue running the custom criteria script.

  5. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

  6. To reuse the script in another test case, call the function from the test case custom criteria.

Assess the Damping Ratio of a Flutter Suppression System

This example uses:

Open Script

Using a custom criteria script, verify that wing oscillations are damped in multiple altitude and airspeed conditions.

The Test and Model

The model uses Simscape™ to simulate a Benchmark Active Controls Technology (BACT) / Pitch and Plunge Apparatus (PAPA) setup. It uses Aerospace Blockset™ to simulate aerodynamic forces on the wing.

The test iterates over 16 combinations of Mach and Altitude. The test case uses custom criteria with Curve Fitting Toolbox™ to find the peaks of the wing pitch, and determine the damping ratio. If the damping ratio is not greater than zero, the assessment fails.

Running this test case requires:

  • Simulink® Test™

  • Simscape Multibody™

  • Aerospace Blockset™

  • Curve Fitting Toolbox™

Click the Open File button to open the test file.

In the Test Browser, select Altitude and mach iterations. Open the model by clicking the arrow next to Model in the System Under Test section.

Custom Criteria Script

The test case custom criteria uses this script to verify that the damping ratio is greater than zero.

% Get time and data for pitch
Time = test.sltest_simout.get('sigsOut').get('pitch').Values.Time(1:15000);
Data = test.sltest_simout.get('sigsOut').get('pitch').Values.Data(1:15000);

% Find peaks
[~, peakIds] = findpeaks(Data,'minpeakheight', 0.002, 'minpeakdistance', 50);
peakTime= Time(peakIds);
peakPos = Data(peakIds);
rn = peakPos(1)./peakPos(2:end);
L = 1:length(rn);

% Do curve fitting
fittedModel = exponentialFitAndPlot(L, rn);
delta = fittedModel.d;

% Find damping ratio
dRatio = delta/sqrt((2*pi)^2+delta^2);

% Make sure damping ratio is greater than 0
test.verifyGreaterThan(dRatio,0,'Damping ratio must be greater than 0');

Test Results

Running the test case returns two conditions in which the damping ratio is greater than zero.

The wing pitch plots from iteration 12 and 13 show the difference between a positive damping ratio (iteration 12) and a negative damping ratio (iteration 13).

Custom Criteria Programmatic Interface Example

Open Live Script

This example shows how to set and get custom criteria using the programmatic interface.

Before running this example, temporarily disable warnings that result from verification failures.

warning off Stateflow:Runtime:TestVerificationFailed;
warning off Stateflow:cdr:VerifyDangerousComparison;

Load a Test File and Get Test Case Object

tf = sltest.testmanager.load('AutopilotTestFile.mldatx');

ts = getTestSuiteByName(tf,'Basic Design Test Cases');

tc = getTestCaseByName(ts,'Requirement 1.3 Test');

Create the Custom Criteria Object and Set Criteria

Create the custom criteria object.

tcCriteria = getCustomCriteria(tc)
tcCriteria = 
  CustomCriteria with properties:

     Enabled: 0
    Callback: '% Return value: customCriteria...'

Create the custom criteria expression. This script gets the last value of the signal Phi and verifies that it equals 0.

criteria = ...
    sprintf(['lastPhi = test.SimOut.get(''Signals_Req1_3'')',...
	'.get(''Phi'').Values.Data(end);\n',...
	'test.verifyEqual(lastPhi,0,[''Final: '',num2str(lastPhi),''.'']);'])
criteria = 
    'lastPhi = test.SimOut.get('Signals_Req1_3').get('Phi').Values.Data(end);
     test.verifyEqual(lastPhi,0,['Final: ',num2str(lastPhi),'.']);'

Set and enable the criteria.

tcCriteria.Callback = criteria;
tcCriteria.Enabled = true;

Run the Test Case and Get the Results

Run the test case.

tcResultSet = run(tc);

Get the test case results.

tcResult = getTestCaseResults(tcResultSet);

Get the custom criteria result.

ccResult = getCustomCriteriaResult(tcResult)
ccResult = 
  CustomCriteriaResult with properties:

             Outcome: Failed
    DiagnosticRecord: [1x1 sltest.testmanager.DiagnosticRecord]

Restore warnings from verification failures.

warning on Stateflow:Runtime:TestVerificationFailed;
warning on Stateflow:cdr:VerifyDangerousComparison;
sltest.testmanager.clearResults
sltest.testmanager.clear
sltest.testmanager.close

Related Topics

Simulink Test Documentation
Support