matlab.perftest.TimeExperiment.limitingSamplingError

Class: matlab.perftest.TimeExperiment
Package: matlab.perftest

Construct time experiment for specified margin of error and confidence level

expand all in page

Syntax

experiment = matlab.perftest.TimeExperiment.limitingSamplingError
experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value)

Description

example

experiment = matlab.perftest.TimeExperiment.limitingSamplingError constructs a time experiment for each test suite element, with the specified statistical objectives (such as margin of error and confidence level). This method returns an instance of FrequentistTimeExperiment. This syntax uses the following defaults to determine the number of sample measurements.

  • Number of warm-up measurements: 4

  • Minimum number of samples: 4

  • Maximum number of samples collected in the event other statistical objectives are not met: 256

  • Objective relative margin of error for samples: 0.05 (5%)

  • Confidence level for samples to be within relative margin of error: 0.95 (95%)

experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value) constructs a time experiment with additional options specified by one or more Name,Value pair arguments. Use this syntax to override the defaults listed above.

Input Arguments

expand all

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: experiment = matlab.perftest.TimeExperiment.limitingSamplingError('RelativeMarginOfError',0.12,'MaxSamples',100) creates a time experiment that collects sample measurements until samples have a relative margin of error of 12%, or until it collects 100 measurements.

Number of warm-up measurements, specified as a nonnegative integer. The value defines the number of times that the test framework exercises the test code to warm it up.

Minimum number of sample measurements, specified as a positive integer. The value defines the minimum number of times that the test framework exercises the test code after any warm-up runs. The test framework exercises the test code at least MinSamples times, regardless of whether the experiment meets the statistical objectives.

Maximum number of sample measurements, specified as a positive integer. The value defines the maximum number of times that the test framework exercises the test code after NumWarmups. If the experiment does not meet the statistical objectives, the testing framework collects up to MaxSamples.

Objective relative margin of error for samples, specified as a positive number.

The testing framework calculates the relative margin of error for a sample X using the equation

relMoE=T∗std(X)mean(X)∗length(X)

where T is the T-score from Student's T distribution using the specified ConfidenceLevel and length(X)-1 degrees of freedom.

Confidence level for the samples to be within the relative margin of error, specified as a number from 0 through 1.

Examples

expand all

In your current working folder, create a class-based test, preallocationTest.m, that compares different methods of preallocation.

classdef preallocationTest < matlab.perftest.TestCase
    methods(Test)
        function testOnes(testCase)
            x = ones(1,1e7);
        end
        function testIndexingWithVariable(testCase)
            id = 1:1e7;
            x(id) = 1;
        end
        function testIndexingOnLHS(testCase)
            x(1:1e7) = 1;
        end
        function testForLoop(testCase)
            for i=1:1e7
                x(i) = 1;
            end
        end
        
    end
end

Create a test suite.

suite = testsuite('preallocationTest');

Construct a time experiment with a variable number of sample measurements, and run the tests.

import matlab.perftest.TimeExperiment
experiment = TimeExperiment.limitingSamplingError;
result = run(experiment,suite);
Running preallocationTest
.......... .......... .......... ..
Done preallocationTest
__________

View the test activity for the first test. Your results might vary.

result(1).TestActivity
ans =

  8×12 table

               Name               Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform           Version                      TestResult                          RunIdentifier            
    __________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    _____________________    ________________________________    ____________________________________

    preallocationTest/testOnes    true      false       false         0.056052       warmup      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.056227       warmup      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.055969       warmup      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.054961       warmup      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.052572       sample      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.051743       sample      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.051709       sample      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12
    preallocationTest/testOnes    true      false       false         0.051256       sample      05-Oct-2018 10:14:15    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    34fa8e1d-e21f-42b5-83bd-fd104ffcec12

For this test, the performance testing framework collected 4 warm-up measurements (the default), and 11 sample measurements. After 11 sample measurements, the performance testing framework satisfied the default statistical objectives.

Construct a time experiment that collects two warm-up measurements and runs the tests a variable number of times to reach a sample mean with a 10% relative margin of error within a 90% confidence level.

experiment = TimeExperiment.limitingSamplingError('NumWarmups',2,...
    'RelativeMarginOfError',0.10, 'ConfidenceLevel', 0.90);
result = run(experiment, suite);
Running preallocationTest
.......... .......... ....
Done preallocationTest
__________

View the test activity for the first test. Your results might vary.

result(1).TestActivity
ans = 
  6×12 table

               Name               Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform           Version                      TestResult                          RunIdentifier            
    __________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    _____________________    ________________________________    ____________________________________

    preallocationTest/testOnes    true      false       false         0.053963       warmup      05-Oct-2018 10:21:31    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1
    preallocationTest/testOnes    true      false       false         0.053086       warmup      05-Oct-2018 10:21:31    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1
    preallocationTest/testOnes    true      false       false         0.052502       sample      05-Oct-2018 10:21:31    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1
    preallocationTest/testOnes    true      false       false          0.05252       sample      05-Oct-2018 10:21:31    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1
    preallocationTest/testOnes    true      false       false         0.052048       sample      05-Oct-2018 10:21:32    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1
    preallocationTest/testOnes    true      false       false         0.052434       sample      05-Oct-2018 10:21:32    MY-HOSTNAME     win64      9.6.0.966561 (R2019a)    [1x1 matlab.unittest.TestResult]    22e4507c-e12c-4cac-8730-aff65e75a2e1

For this test, the performance testing framework collected two warm-up measurements and nine sample measurements. After nine sample measurements, the performance testing framework satisfied the specified statistical objectives.

See Also

|

Topics

Introduced in R2016a

MATLAB Documentation

Support