%% Automate Testing for Highway Lane Following
% This example shows how to assess the functionality of a lane-following
% application by defining scenarios based on requirements, automating
% testing of components and the generated code for those components. The
% components include lane-detection, sensor fusion, decision logic, and
% controls. This example builds on the
% <docid:driving_ug#mw_574c4bc8-4369-4da6-a4d3-aa48e7e31f92 Highway
% Lane Following> example.

% Copyright 2019-2020 The MathWorks, Inc.

%% Introduction
% A highway lane-following system steers a vehicle to travel within a
% marked lane. It also maintains a set velocity or safe distance from a
% preceding vehicle in the same lane. The system typically includes lane
% detection, sensor fusion, decision logic, and controls components.
% System-level simulation is a common technique for assessing functionality
% of the integrated components. Simulations are configured to test
% scenarios based on system requirements. Automatically running these
% simulations enables regression testing to verify system-level
% functionality.
%
% The <docid:driving_ug#mw_574c4bc8-4369-4da6-a4d3-aa48e7e31f92
% Highway Lane Following> example showed how to simulate a system-level
% model for lane-following. This example shows how to automate testing that
% model against multiple scenarios using Simulink Test&trade;. The
% scenarios are based on system-level requirements. In this example, you
% will:
%
% # *Review requirements:* The requirements describe system-level test
% conditions. Simulation test scenarios are created to represent these
% conditions.
% # *Review the test bench model:* Review the system-level lane-following
% test bench model that contains metric assessments. These metric
% assessments integrate the test bench model with Simulink Test for the
% automated testing.
% # *Disable runtime visualizations:* Runtime visualizations are disabled
% to reduce execution time for the automated testing.
% # *Automate testing:* A test manager is configured to simulate each test
% scenario, assess success criteria, and report results. The results are
% explored dynamically in the test manager and exported to a PDF for
% external reviewers.
% # *Automate testing with generated code:* The lane detection, sensor
% fusion, decision logic, and controls components are configured to
% generate C++ code. The automated testing is run on the generated code to
% verify expected behavior.
% # *Automate testing in parallel:* Overall execution time for running the
% tests is reduced using parallel computing on a multi-core computer.
%
% Testing the system-level model requires a photorealistic simulation
% environment. In this example, you enable system-level simulation through
% integration with the Unreal Engine from Epic Games&reg;. The 3D
% simulation environment requires a Windows&reg; 64-bit platform.
%
if ~ispc
    error("The 3D simulation environment requires a Windows 64-bit platform");
end
%%
% To ensure reproducibility of the simulation results, set the random seed. 
rng(0);

%% Review Requirements
% Simulink Requirements&trade; lets you author, analyze, and manage requirements
% within Simulink. This example contains ten test scenarios, with
% high-level testing requirements defined for each scenario. Open the
% requirement set. 
%%
open('HighwayLaneFollowingTestRequirements.slreqx')
%%
% Alternatively, you can also open the file from the
% *Requirements* tab of the Requirements Manager app in Simulink.
%%
% <<../LF_SLRequirements.PNG>>
% 
%%
% Each row in this file specifies the requirements in textual and graphical
% formats for testing the lane-following system for a test
% scenario. The scenarios with the *scenario_LF_* prefix enable you to test
% lane-detection and lane-following algorithms without obstruction by other
% vehicles. The scenarios with the *scenario_LFACC_* prefix enable you to
% test lane-detection, lane-following, and ACC behavior with other vehicles
% on the road.
%%
% # |scenario_LF_01_Straight_RightLane| &mdash; Straight road scenario
% with ego vehicle in right lane.
% # |scenario_LF_02_Straight_LeftLane| &mdash; Straight road scenario
% with ego vehicle in left lane.
% # |scenario_LF_03_Curve_LeftLane| &mdash; Curved road scenario with
% ego vehicle in left lane.
% # |scenario_LF_04_Curve_RightLane| &mdash; Curved road scenario with
% ego vehicle in right lane.
% # |scenario_LFACC_01_Curve_DecelTarget| &mdash; Curved road scenario
% with a decelerating lead vehicle in ego lane.
% # |scenario_LFACC_02_Curve_AutoRetarget| &mdash; Curved road scenario
% with changing lead vehicles in ego lane. This scenario tests the ability
% of the ego vehicle to retarget to a new lead vehicle while driving along
% a curve.
% # |scenario_LFACC_03_Curve_StopnGo| &mdash; Curved road scenario with
% a lead vehicle slowing down in ego lane.
% # |scenario_LFACC_04_Curve_CutInOut| &mdash; Curved road scenario
% with a lead car cutting into ego lane.
% # |scenario_LFACC_05_Curve_CutInOut_TooClose| &mdash; Curved road
% scenario with a lead car cutting aggressively into the ego lane.
% # |scenario_LFACC_06_Straight_StopandGoLeadCar| &mdash; Straight road
% scenario with a lead vehicle that breaks down in ego lane.
% 
% These requirements are implemented as test scenarios with the same
% names as the scenarios used in the |HighwayLaneFollowingTestBench| model.
%% Review Test Bench Model
% This example reuses the |HighwayLaneFollowingTestBench| model from the
% <docid:driving_ug#mw_574c4bc8-4369-4da6-a4d3-aa48e7e31f92 Highway
% Lane Following> example. Open the test bench model.
%%
open_system("HighwayLaneFollowingTestBench");
%%
% This test bench model has *Simulation 3D Scenario*, *Vision Detector
% Variant*, *Forward Vehicle Sensor Fusion*, *Lane Following Decision and
% Controller* and *Vehicle Dynamics* components. 
% 
% This test bench model is configured using the
% |helperSLHighwayLaneFollowingSetup| script. This setup script takes two
% inputs: |scenarioName| and |visionVariantAlgorithm|. |scenarioName| can
% be any one of the previously described test scenarios.
% |visionVariantAlgorithm| can be either |ProbabilisticDetectionSensor| or
% |VisionProcessingAlgorithm|. The |ProbabilisticDetectionSensor| variant
% enables you to test integration of the control algorithm in the 3D
% simulation environment, without also integrating the vision processing
% algorithm. The |VisionProcessingAlgorithm| variant enables you to test
% integration of the control algorithm and vision processing algorithm in
% the 3D simulation environment. To run the setup script, use code:
%%
scenarioName = "scenario_LFACC_03_Curve_StopnGo";
visionVariantAlgorithm = "VisionProcessingAlgorithm";
helperSLHighwayLaneFollowingSetup(scenarioName,visionVariantAlgorithm);
%%
% You can now simulate the model and visualize the results. For more
% details on the analysis of the simulation results and the design of
% individual components in the test bench model, see the
% <docid:driving_ug#mw_574c4bc8-4369-4da6-a4d3-aa48e7e31f92 Highway
% Lane Following> example.
%
% In this example, the focus is more on automating the simulation runs for
% this test bench model using Simulink Test for the different test
% scenarios. The *Metrics Assessment* subsystem enables integration of
% system-level metric evaluations with Simulink Test. This subsystem uses
% <docid:simulink_ref#f6-1667783 Check Static Range> blocks for this
% integration. Open the *Metrics Assessment* subsystem.
%%
open_system("HighwayLaneFollowingTestBench/Metrics Assessment");
%%
% In this example, four metrics are used to assess the lane-following
% system.  
%
% * *Verify Lateral Deviation*: Verifies that the lateral deviation from
% the centerline of the lane is within prescribed thresholds for the
% corresponding scenario. Prescribed thresholds are defined while authoring
% the test scenario.
% 
% * *Verify In Lane*: Verifies that the ego vehicle is following one of the
% lanes on the road throughout the simulation.
% 
% * *Verify Time gap*: Verifies that the time gap between the ego vehicle
% and the lead vehicle is above 0.8 seconds. The time gap between the two
% vehicles is defined as the ratio of the calculated headway distance to
% the ego vehicle velocity.
%
% * *Verify No Collision*: Verifies that the ego vehicle does not collide
% with the lead vehicle at any point during the simulation.
%% Disable Runtime Visualizations
% The system-level test bench model visualizes
% intermediate outputs during the simulation for the analysis of different
% components in the model. These visualizations are not required when the
% tests are automated. You can reduce execution time for the automated
% testing by disabling them.
%%
% Disable runtime visualizations for the *Lane Marker Detector* subsystem.
%%
load_system('LaneMarkerDetector');
blk = 'LaneMarkerDetector/Lane Marker Detector';
set_param(blk,'EnableDisplays','off');
%%
% Disable runtime visualizations for the *Vision Vehicle Detector* subsystem.
%%
blk = 'HighwayLaneFollowingTestBench/Vision Detector Variant/Vision Processing Algorithm/Vision Vehicle Detector';
set_param(blk,'EnableDisplay','off');
%%
% Configure the <docid:driving_ref#mw_96a521fd-316f-497b-bc01-b2c5f4083563
% Simulation 3D Scene Configuration> block to run the Unreal
% Engine in headless mode, where the 3D simulation window is disabled.
%%
blk = 'HighwayLaneFollowingTestBench/Simulation 3D Scenario/Simulation 3D Scene Configuration';
set_param(blk,'EnableWindow','off');

%% Automate Testing 
% The Test Manager is configured to automate the testing of the
% lane-following application. Open the
% |HighwayLaneFollowingMetricAssessments.mldatx| test file in the Test
% Manager.
sltestmgr;
sltest.testmanager.load('HighwayLaneFollowingMetricAssessments.mldatx');
%%
%
% <<../LF_SLTest.PNG>>
%%
% Observe the populated test cases that were authored previously in this
% file. Each test case is linked to the corresponding requirement in the
% Requirements Editor for traceability. These tests are configured to run
% with the |VisionProcessingAlgorithm| variant. You can also configure
% tests to run with the |ProbabilisticDetectionSensor| variant by modifying
% the |visionVariant| variable in the test suite |SETUP| callback.
%
% Each test case uses the |POST-LOAD| callback to run the setup script with
% appropriate inputs and to configure the output video file name. After the
% simulation of the test case, it invokes
% |helperGenerateFilesForLaneFollowingReport| from the |CLEAN-UP| callback
% to generate the plots explained in the
% <docid:driving_ug#mw_574c4bc8-4369-4da6-a4d3-aa48e7e31f92 Highway
% Lane Following> example.
% 
% *Run and explore results for a single test scenario:*
%
% To test the system-level model with the |scenario_LFACC_03_Curve_StopnGo|
% test scenario from Simulink Test, use this code:
%%
testFile = sltest.testmanager.TestFile('HighwayLaneFollowingMetricAssessments.mldatx');
testSuite = getTestSuiteByName(testFile,'Test Scenarios');
testCase = getTestCaseByName(testSuite,'scenario_LFACC_03_Curve_StopnGo');
resultObj = run(testCase);

%% 
% To generate a report after the simulation, use this code:
sltest.testmanager.report(resultObj,'Report.pdf',...,
    'Title','Highway Lane Following',...
    'IncludeMATLABFigures',true,...
    'IncludeErrorMessages',true,...
    'IncludeTestResults',0,'LaunchReport',true);
%%
% Examine the |Report.pdf|. Observe that the *Test environment* section
% shows the platform on which the test is run and the MATLAB&reg; version
% used for testing. The *Summary* section shows the outcome of the test and
% duration of the simulation in seconds. The *Results* section shows
% pass/fail results based on the assessment criteria. This section also
% shows the plots logged from the
% |helperGenerateFilesForLaneFollowingReport| function.
%
%%
% *Run and explore results for all test scenarios:*
%
% You can simulate the system for all the tests by using
% |sltest.testmanager.run|. Alternatively, you can simulate the system by
% clicking *Play* in the Test Manager app.
% 
% After completion of the test simulations, the results for all the tests
% can be viewed in the *Results and Artifacts* tab of the Test Manager. For
% each test case, the <docid:simulink_ref#f6-1667783 Check Static Range>
% blocks in the model are associated with the Test Manager to visualize
% overall pass/fail results.
% 
% <<../LF_SLTest_Results.PNG>>
%
% You can find the generated report in current working directory. This
% report contains a detailed summary of pass/fail statuses and plots for
% each test case.
%
% <<../LF_SLTest_Report.PNG>>
%
% *Verify test status in Requirements Editor:*
%
% Open the Requirements Editor and select *Display*. Then, select
% *Verification Status* to see a verification status summary for each
% requirement. Green and red bars indicate the pass/fail status of
% simulation results for each test.
%
% <<../LF_SLTest_Verification.PNG>> 
%% Automate Testing with Generated Code
% The |HighwayLaneFollowingTestBench| model enables integrated testing of
% *Lane Marker Detector*, *Forward Vehicle Sensor Fusion*, and *Lane
% Following Decision Logic and Controller* components. It is often helpful
% to perform regression testing of these components through
% software-in-the-loop (SIL) verification. If you have a Simulink
% Coder&trade; license, then you can generate code for these components.
% This workflow lets you verify that the generated code produces expected
% results that match the system-level requirements throughout simulation.
%%
% Set *Lane Marker Detector* to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Vision Detector Variant/Vision Processing Algorithm/Lane Marker Detector'; 
set_param(model,'SimulationMode','Software-in-the-loop');
%%
% Set *Forward Vehicle Sensor Fusion* to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Forward Vehicle Sensor Fusion'; 
set_param(model,'SimulationMode','Software-in-the-loop');
%%
% Set *Lane Following Decision Logic and Controller* to run in
% Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Lane Following Decision Logic and Controller';
set_param(model,'SimulationMode','Software-in-the-loop');
%%
% Now, run |sltest.testmanager.run| to simulate the system for all the test
% scenarios. After the completion of tests, review the plots and results
% in the generated report. 
%% Automate Testing in Parallel
% If you have a Parallel Computing Toolbox&trade; license, then you can
% configure Test Manager to execute tests in parallel using a parallel
% pool.  To run tests in parallel, save the models after disabling the
% runtime visualizations using |save_system('LaneMarkerDetector')| and
% |save_system('HighwayLaneFollowingTestBench')|. Test Manager uses the
% default Parallel Computing Toolbox cluster and executes tests only on the
% local machine. Running tests in parallel can speed up execution and
% decrease the amount of time it takes to get test results. For more
% information on how to configure tests in parallel from the Test Manager,
% see <docid:sltest_ug#bu8a6bv Run Tests Using Parallel Execution>.