Test Performance Using Scripts or Functions

This example shows how to create a script or function-based performance test that times the preallocation of a vector using four different approaches.

Write Performance Test

Create a performance test in a file, preallocationTest.m, in your current working folder. In this example, you can choose to use either the following script-based test or the function-based test. The output in this example is for the function-based test. If you use the script-based test, then your test names will be different.

Script-Based Performance Test	Function-Based Performance Test
vectorSize = 1e7; %% Ones Function x = ones(1,vectorSize); %% Indexing With Variable id = 1:vectorSize; x(id) = 1; %% Indexing On LHS x(1:vectorSize) = 1; %% For Loop for i=1:vectorSize x(i) = 1; end	function tests = preallocationTest tests = functiontests(localfunctions); end function testOnes(testCase) vectorSize = getSize(); x = ones(1,vectorSize()); end function testIndexingWithVariable(testCase) vectorSize = getSize(); id = 1:vectorSize; x(id) = 1; end function testIndexingOnLHS(testCase) vectorSize = getSize(); x(1:vectorSize) = 1; end function testForLoop(testCase) vectorSize = getSize(); for i=1:vectorSize x(i) = 1; end end function vectorSize = getSize() vectorSize = 1e7; end

Script-Based Performance Test

Function-Based Performance Test

vectorSize = 1e7;

%% Ones Function
x = ones(1,vectorSize);

%% Indexing With Variable
id = 1:vectorSize;
x(id) = 1;

%% Indexing On LHS
x(1:vectorSize) = 1;

%% For Loop
for i=1:vectorSize
    x(i) = 1;
end

function tests = preallocationTest
tests = functiontests(localfunctions);
end

function testOnes(testCase)
vectorSize = getSize();
x = ones(1,vectorSize());
end

function testIndexingWithVariable(testCase)
vectorSize = getSize();
id = 1:vectorSize;
x(id) = 1;
end

function testIndexingOnLHS(testCase)
vectorSize = getSize();
x(1:vectorSize) = 1;
end

function testForLoop(testCase)
vectorSize = getSize();
for i=1:vectorSize
    x(i) = 1;
end
end

function vectorSize = getSize()
vectorSize = 1e7;
end

Run Performance Test

Run the performance test using runperf.

results = runperf('preallocationTest.m')

Running preallocationTest
.......... .......... .......... ..
Done preallocationTest
__________


results = 

  1×4 TimeResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   4 Valid, 0 Invalid.
   10.2561 seconds testing time.

The results variable is a 1-by-4 TimeResult array. Each element in the array corresponds to one of the tests defined in the code section in preallocationTest.m.

Display Test Results

Display the measurement results for the second test. Your results might vary.

results(2)

ans = 

  TimeResult with properties:

            Name: 'preallocationTest/testIndexingWithVariable'
           Valid: 1
         Samples: [4×7 table]
    TestActivity: [8×12 table]

Totals:
   1 Valid, 0 Invalid.
   1.2274 seconds testing time.

As indicated by the size of the TestActivity property, the performance testing framework collected 8 measurements. This number of measurements includes four measurements to warm up the code. The Samples property excludes warm-up measurements.

Display the sample measurements for the second test.

results(2).Samples

>> results(2).Samples

ans =

  4×7 table

                       Name                       MeasuredTime         Timestamp             Host        Platform                     Version                                 RunIdentifier            
    __________________________________________    ____________    ____________________    ___________    ________    __________________________________________    ____________________________________

    preallocationTest/testIndexingWithVariable      0.15271       24-Jun-2019 16:13:33    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    9a6ee247-b49d-4e29-8b63-ba13c2eead27
    preallocationTest/testIndexingWithVariable      0.15285       24-Jun-2019 16:13:33    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    9a6ee247-b49d-4e29-8b63-ba13c2eead27
    preallocationTest/testIndexingWithVariable      0.15266       24-Jun-2019 16:13:33    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    9a6ee247-b49d-4e29-8b63-ba13c2eead27
    preallocationTest/testIndexingWithVariable      0.15539       24-Jun-2019 16:13:34    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    9a6ee247-b49d-4e29-8b63-ba13c2eead27

Compute Statistics for Single Test Element

Display the mean measured time for the second test. To exclude data collected in the warm-up runs, use the values in the Samples field.

sampleTimes = results(2).Samples.MeasuredTime;
meanTest2 = mean(sampleTimes)

meanTest2 =

    0.1534

The performance testing framework collected four sample measurements for the second test. The test took an average of 0.1534 second.

Compute Statistics for All Test Elements

Determine the average time for all the test elements. The preallocationTest test includes four different methods for allocating a vector of ones. Compare the time for each method (test element).

Since the performance testing framework returns a Samples table for each test element, concatenate all these tables into one table. Then group the rows by test element Name, and compute the mean MeasuredTime for each group.

fullTable = vertcat(results.Samples);
summaryStats = varfun(@mean,fullTable,...
    'InputVariables','MeasuredTime','GroupingVariables','Name')

summaryStats =

  4×3 table

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testOnes                        4             0.041072     
    preallocationTest/testIndexingWithVariable        4               0.1534     
    preallocationTest/testIndexingOnLHS               4              0.04677     
    preallocationTest/testForLoop                     4               1.0343

Change Statistical Objectives and Rerun Tests

Change the statistical objectives defined by the runperf function by constructing and running a time experiment. Construct a time experiment with measurements that reach a sample mean with an 8% relative margin of error within a 97% confidence level.

Construct an explicit test suite.

suite = testsuite('preallocationTest');

Construct a time experiment with a variable number of sample measurements, and run the tests.

import matlab.perftest.TimeExperiment
experiment = TimeExperiment.limitingSamplingError('NumWarmups',2,...
    'RelativeMarginOfError',0.08, 'ConfidenceLevel', 0.97);
resultsTE = run(experiment,suite);

Running preallocationTest
.......... .......... ....
Done preallocationTest
__________

Compute the statistics for all the test elements.

fullTableTE = vertcat(resultsTE.Samples);
summaryStatsTE = varfun(@mean,fullTableTE,...
    'InputVariables','MeasuredTime','GroupingVariables','Name')

summaryStatsTE =

  4×3 table

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testOnes                        4             0.040484     
    preallocationTest/testIndexingWithVariable        4              0.15187     
    preallocationTest/testIndexingOnLHS               4             0.046224     
    preallocationTest/testForLoop                     4               1.0262

Documentation