Introduction

A highway lane-following system steers a vehicle to travel within a marked lane. It also maintains a set velocity or safe distance from a preceding vehicle in the same lane. The system typically includes lane detection, sensor fusion, decision logic, and controls components. System-level simulation is a common technique for assessing functionality of the integrated components. Simulations are configured to test scenarios based on system requirements. Automatically running these simulations enables regression testing to verify system-level functionality.

The Highway Lane Following example showed how to simulate a system-level model for lane-following. This example shows how to automate testing that model against multiple scenarios using Simulink Test™. The scenarios are based on system-level requirements. In this example, you will:

Testing the system-level model requires a photorealistic simulation environment. In this example, you enable system-level simulation through integration with the Unreal Engine from Epic Games®. The 3D simulation environment requires a Windows® 64-bit platform.

if ~ispc
    error("The 3D simulation environment requires a Windows 64-bit platform");
end

To ensure reproducibility of the simulation results, set the random seed.

rng(0);

Review Requirements

Simulink Requirements™ lets you author, analyze, and manage requirements within Simulink. This example contains ten test scenarios, with high-level testing requirements defined for each scenario. Open the requirement set.

open('HighwayLaneFollowingTestRequirements.slreqx')

Alternatively, you can also open the file from the Requirements tab of the Requirements Manager app in Simulink.

Each row in this file specifies the requirements in textual and graphical formats for testing the lane-following system for a test scenario. The scenarios with the scenario_LF_ prefix enable you to test lane-detection and lane-following algorithms without obstruction by other vehicles. The scenarios with the scenario_LFACC_ prefix enable you to test lane-detection, lane-following, and ACC behavior with other vehicles on the road.

These requirements are implemented as test scenarios with the same names as the scenarios used in the HighwayLaneFollowingTestBench model.

Review Test Bench Model

This example reuses the HighwayLaneFollowingTestBench model from the Highway Lane Following example. Open the test bench model.

open_system("HighwayLaneFollowingTestBench");

This test bench model has Simulation 3D Scenario, Vision Detector Variant, Forward Vehicle Sensor Fusion, Lane Following Decision and Controller and Vehicle Dynamics components.

This test bench model is configured using the helperSLHighwayLaneFollowingSetup script. This setup script takes two inputs: scenarioName and visionVariantAlgorithm. scenarioName can be any one of the previously described test scenarios. visionVariantAlgorithm can be either ProbabilisticDetectionSensor or VisionProcessingAlgorithm. The ProbabilisticDetectionSensor variant enables you to test integration of the control algorithm in the 3D simulation environment, without also integrating the vision processing algorithm. The VisionProcessingAlgorithm variant enables you to test integration of the control algorithm and vision processing algorithm in the 3D simulation environment. To run the setup script, use code:

scenarioName = "scenario_LFACC_03_Curve_StopnGo";
visionVariantAlgorithm = "VisionProcessingAlgorithm";
helperSLHighwayLaneFollowingSetup(scenarioName,visionVariantAlgorithm);

You can now simulate the model and visualize the results. For more details on the analysis of the simulation results and the design of individual components in the test bench model, see the Highway Lane Following example.

In this example, the focus is more on automating the simulation runs for this test bench model using Simulink Test for the different test scenarios. The Metrics Assessment subsystem enables integration of system-level metric evaluations with Simulink Test. This subsystem uses Check Static Range (Simulink) blocks for this integration. Open the Metrics Assessment subsystem.

open_system("HighwayLaneFollowingTestBench/Metrics Assessment");

In this example, four metrics are used to assess the lane-following system.

Disable Runtime Visualizations

The system-level test bench model visualizes intermediate outputs during the simulation for the analysis of different components in the model. These visualizations are not required when the tests are automated. You can reduce execution time for the automated testing by disabling them.

Disable runtime visualizations for the Lane Marker Detector subsystem.

load_system('LaneMarkerDetector');
blk = 'LaneMarkerDetector/Lane Marker Detector';
set_param(blk,'EnableDisplays','off');

Disable runtime visualizations for the Vision Vehicle Detector subsystem.

blk = 'HighwayLaneFollowingTestBench/Vision Detector Variant/Vision Processing Algorithm/Vision Vehicle Detector';
set_param(blk,'EnableDisplay','off');

Configure the Simulation 3D Scene Configuration block to run the Unreal Engine in headless mode, where the 3D simulation window is disabled.

blk = 'HighwayLaneFollowingTestBench/Simulation 3D Scenario/Simulation 3D Scene Configuration';
set_param(blk,'EnableWindow','off');

Automate Testing

The Test Manager is configured to automate the testing of the lane-following application. Open the HighwayLaneFollowingMetricAssessments.mldatx test file in the Test Manager.

sltestmgr;
sltest.testmanager.load('HighwayLaneFollowingMetricAssessments.mldatx');

Observe the populated test cases that were authored previously in this file. Each test case is linked to the corresponding requirement in the Requirements Editor for traceability. These tests are configured to run with the VisionProcessingAlgorithm variant. You can also configure tests to run with the ProbabilisticDetectionSensor variant by modifying the visionVariant variable in the test suite SETUP callback.

Each test case uses the POST-LOAD callback to run the setup script with appropriate inputs and to configure the output video file name. After the simulation of the test case, it invokes helperGenerateFilesForLaneFollowingReport from the CLEAN-UP callback to generate the plots explained in the Highway Lane Following example.

Run and explore results for a single test scenario:

To test the system-level model with the scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test, use this code:

testFile = sltest.testmanager.TestFile('HighwayLaneFollowingMetricAssessments.mldatx');
testSuite = getTestSuiteByName(testFile,'Test Scenarios');
testCase = getTestCaseByName(testSuite,'scenario_LFACC_03_Curve_StopnGo');
resultObj = run(testCase);

To generate a report after the simulation, use this code:

sltest.testmanager.report(resultObj,'Report.pdf',...,
    'Title','Highway Lane Following',...
    'IncludeMATLABFigures',true,...
    'IncludeErrorMessages',true,...
    'IncludeTestResults',0,'LaunchReport',true);

Examine the Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB® version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass/fail results based on the assessment criteria. This section also shows the plots logged from the helperGenerateFilesForLaneFollowingReport function.

Run and explore results for all test scenarios:

You can simulate the system for all the tests by using sltest.testmanager.run. Alternatively, you can simulate the system by clicking Play in the Test Manager app.

After completion of the test simulations, the results for all the tests can be viewed in the Results and Artifacts tab of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass/fail results.

You can find the generated report in current working directory. This report contains a detailed summary of pass/fail statuses and plots for each test case.

Verify test status in Requirements Editor:

Open the Requirements Editor and select Display. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass/fail status of simulation results for each test.

Automate Testing with Generated Code

The HighwayLaneFollowingTestBench model enables integrated testing of Lane Marker Detector, Forward Vehicle Sensor Fusion, and Lane Following Decision Logic and Controller components. It is often helpful to perform regression testing of these components through software-in-the-loop (SIL) verification. If you have a Simulink Coder™ license, then you can generate code for these components. This workflow lets you verify that the generated code produces expected results that match the system-level requirements throughout simulation.

Set Lane Marker Detector to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Vision Detector Variant/Vision Processing Algorithm/Lane Marker Detector';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Forward Vehicle Sensor Fusion to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Forward Vehicle Sensor Fusion';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Lane Following Decision Logic and Controller to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Lane Following Decision Logic and Controller';
set_param(model,'SimulationMode','Software-in-the-loop');

Now, run sltest.testmanager.run to simulate the system for all the test scenarios. After the completion of tests, review the plots and results in the generated report.

Automate Testing in Parallel

If you have a Parallel Computing Toolbox™ license, then you can configure Test Manager to execute tests in parallel using a parallel pool. To run tests in parallel, save the models after disabling the runtime visualizations using save_system('LaneMarkerDetector') and save_system('HighwayLaneFollowingTestBench'). Test Manager uses the default Parallel Computing Toolbox cluster and executes tests only on the local machine. Running tests in parallel can speed up execution and decrease the amount of time it takes to get test results. For more information on how to configure tests in parallel from the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).