%% Evaluate Deep Learning Experiments by Using Metric Functions
% This example shows how to use metric functions to evaluate the results of
% an experiment. By default, when you run a built-in training experiment,
% *<docid:nnet_ref#mw_11ce840f-eed7-4975-9c28-baeebd79f092 Experiment
% Manager>* computes the loss, accuracy (for classification experiments),
% and root mean squared error (for regression experiments) for each trial
% in your experiment. To compute other measures, create your own metric
% function. For example, you can define metric functions to:
% 
% * Test the prediction performance of a trained network.
% * Evaluate the training progress by computing the slope of the
% validation loss over the final epoch.
% * Display the size of the network used in an experiment that uses
% different network architectures for each trial.
%
% When each trial finishes training, Experiment Manager evaluates the
% metric functions and displays their values in the results table.
%
% In this example, you train a network to classify images of handwritten
% digits. Two metric functions determine how well the trained network
% identifies the images of the numerals one and seven. For more information
% on using Experiment Manager to train a network for image classification,
% see <docid:nnet_ref#mw_f45a80f6-7ebf-45a1-bbef-865ad158611c Sweep
% Hyperparameters to Train a Classification Network>.

% Copyright 2019-2021 The MathWorks, Inc. 

%% Define Metric Functions
% Add a metric function to a built-in training experiment.
%
% 1. In the *Experiment* pane, under *Metrics*, click *Add*. 
%
% 2. In the *Add metric* dialog box, enter a name for the metric function
% and click *OK*. If you enter the name of a function that already exists
% in the project, Experiment Manager adds it to the experiment. Otherwise,
% Experiment Manager creates a function defined by a default template.
%
% <<../exp_mgr_metric_dialog.png>>
%
% 3. Select the name of the metric function and click *Edit*. The metric
% function opens in MATLAB(R) Editor.
%
% The input to a metric function is a structure with three fields:
% 
% * |trainedNetwork| is the |SeriesNetwork| object or |DAGNetwork| object
% returned by the <docid:nnet_ref#bu6sn4c trainNetwork> function. For more
% information, see <docid:nnet_ref#bu6sn4c-trainedNet Trained Network>.
% * |trainingInfo| is a structure containing the training information
% returned by the |trainNetwork| function. For more information, see
% <docid:nnet_ref#bu6sn4c-traininfo Training Information>.
% * |parameters| is a structure with fields from the hyperparameter table.
%
% The output of a custom metric function must be a scalar number, a logical
% value, or a string.

%% Open Experiment
% First, open the example. Experiment Manager loads a project with a
% preconfigured experiment that you can inspect and run. To open the
% experiment, in the *Experiment Browser* pane, double-click the name of
% the experiment (|ClassificationExperiment|).
% 
% <<../exp_mgr_metric_config.png>>

% Project Name: 'DigitClassificationWithMetricsProject'

%%
% Built-in training experiments consist of a description, a table of
% hyperparameters, a setup function, and a collection of metric functions
% to evaluate the results of the experiment. For more information, see
% <docid:nnet_ref#mw_92d8f99c-f283-4bc5-a5d7-7d779c4831f7 Configure Deep
% Learning Experiment>.
%
% The *Description* field contains a textual description of the experiment.
% For this example, the description is:
%
%  Classification of digits, evaluating results by using metric functions:
%  * OnesAsSevens returns the percentage of 1s misclassified as 7s.
%  * SevensAsOnes returns the percentage of 7s misclassified as 1s.
%
% The *Hyperparameters* section specifies the strategy (|Exhaustive Sweep|)
% and hyperparameter values to use for the experiment. When you run the
% experiment, Experiment Manager trains the network using every combination
% of hyperparameter values specified in the hyperparameter table. This
% example uses the hyperparameters |InitialLearnRate| and |Momentum|.
%
% The *Setup Function* configures the training data, network architecture,
% and training options for the experiment. The input to the setup function
% is a structure with fields from the hyperparameter table. The setup
% function returns three outputs that you use to train a network for image
% classification problems. In this example, the setup function has three
% sections.
% 
% * *Load Training Data* defines image datastores containing the training
% and validation data. This example loads images from the Digits data set.
% For more information on this data set, see
% <docid:nnet_ug#mw_d3130147-29e6-47eb-b6f3-9a11a4c8b659 Image Data Sets>.
% 
%   digitDatasetPath = fullfile(toolboxdir("nnet"), ...
%       "nndemos","nndatasets","DigitDataset");
%   imdsTrain = imageDatastore(digitDatasetPath, ...
%       IncludeSubfolders=true, ...
%       LabelSource="foldernames");
% 
%   numTrainingFiles = 750;
%   [imdsTrain,imdsValidation] = splitEachLabel(imdsTrain,numTrainingFiles);
% 
% * *Define Network Architecture* defines the architecture for a
% convolutional neural network for deep learning classification. This
% example uses the default classification network provided by the setup
% function template.
% 
%   inputSize = [28 28 1];
%   numClasses = 10;
%   layers = [
%       imageInputLayer(inputSize)
%       convolution2dLayer(5,20)
%       batchNormalizationLayer
%       reluLayer
%       fullyConnectedLayer(numClasses)
%       softmaxLayer
%       classificationLayer];
% 
% * *Specify Training Options* defines a <docid:nnet_ref#bu59f0q
% trainingOptions> object for the experiment. The example loads the values
% for the training options |'InitialLearnRate'| and |'Momentum'| from the
% hyperparameter table.
%
%   options = trainingOptions("sgdm", ...
%       MaxEpochs=5, ... 
%       ValidationData=imdsValidation, ...
%       ValidationFrequency=30, ...
%       InitialLearnRate=params.InitialLearnRate, ...
%       Momentum=params.Momentum, ...
%       Verbose=false);
% 
% To inspect the setup function, under *Setup Function*, click *Edit*. The
% setup function opens in MATLAB(R) Editor. In addition, the code for the
% setup function appears in <#7 Appendix 1> at the end of this example.
%
% The *Metrics* section specifies optional functions that evaluate the
% results of the experiment. Experiment Manager evaluates these functions
% each time it finishes training the network. To inspect a metric function,
% select the name of the metric function and click *Edit*. The metric
% function opens in MATLAB Editor.
%
% This example includes two metric functions.
%
% * |OnesAsSevens| returns the percentage of images of the numeral one that
% the trained network misclassifies as sevens.
% * |SevensAsOnes| returns the percentage of images of the numeral seven
% that the trained network misclassifies as ones.
%
% Each of these functions uses the trained network to classify the entire
% Digits data set. Then, the functions determine the number of images for
% which the actual label and the predicted label disagree. For example, the
% function |OnesAsSevens| computes the number of images with an actual
% label of |'1'| and a predicted label of |'7'|. Similarly, the function
% |SevensAsOnes| computes the number of images with an actual label of
% |'7'| and a predicted label of |'1'|. The code for these metric functions
% appears in <#8 Appendix 2> and <#9 Appendix 3> at the end of this
% example.

%% Run Experiment
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function six times. Each trial uses a different
% combination of hyperparameter values. By default, Experiment Manager runs
% one trial at a time. If you have Parallel Computing Toolbox(TM), you can
% run multiple trials at the same time. For best results, before you run
% your experiment, start a parallel pool with as many workers as GPUs. For
% more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel> and
% <docid:distcomp_ug#mw_57e04559-0b60-42d5-ad55-e77ec5f5865f GPU Support by
% Release>.
%
% * To run one trial of the experiment at a time, on the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available.
%
% A table of results displays the metric function values for each trial.
% 
% <<../exp_mgr_metric_results.png>>
% 

%% Evaluate Results
% To find the best result for your experiment, sort the table of results.
% For example, find the trial with the smallest number of misclassified
% ones.
% 
% # Point to the *OnesAsSevens* column. 
% # Click the triangle icon.
% # Select *Sort in Ascending Order*.
%
% <<../exp_mgr_metric_results_sorted_1as7.png>>
%
% Similarly, find the trial with the smallest number of misclassified
% sevens by opening the drop-down menu for the *SevensAsOnes* column and
% selecting *Sort in Ascending Order*.
%
% <<../exp_mgr_metric_results_sorted_7as1.png>>
%
% If no single trial minimizes both metric functions simultaneously,
% consider giving preference to a trial that ranks well for each metric.
% For instance, in these results, trial 5 ranks as one of the top three
% trials for each metric function.
%
% To record observations about the results of your experiment, add an
% annotation. 
%
% # In the results table, right-click the *OnesAsSevens* cell of the best
% trial.
% # Select *Add Annotation*. 
% # In the *Annotations* pane, enter your observations in the text box.
% # Repeat the previous steps for the *SevensAsOnes* cell.
%
% <<../exp_mgr_metric_annotation.png>>
%
% For more information, see
% <docid:nnet_ref#mw_ffdb175d-cd27-4954-8fa6-e006dbb6dfd0 Sort, Filter, and
% Annotate Experiment Results>.

%% Close Experiment
% In the *Experiment Browser* pane, right-click the name of the project and
% select *Close Project*. Experiment Manager closes all of the experiments
% and results contained in the project.

%% Appendix 1: Setup Function
% This function configures the training data, network architecture, and
% training options for the experiment.
% 
% *Input*
%
% * |params| is a structure with fields from the Experiment Manager
% hyperparameter table.
%
% *Output*
%
% * |imdsTrain| is an image datastore for the training data.
% * |layers| is a layer graph that defines the neural network architecture.
% * |options| is a |trainingOptions| object.
%
% <include>exp_mgr_metric_setup_function.m</include>

%% Appendix 2: Find Ones Misclassified as Sevens
% This function determines the number of ones that are misclassified as sevens.
%
% <include>exp_mgr_metric_OnesAsSevens.m</include>

%% Appendix 3: Find Sevens Misclassified as Ones
% This function determines the number of sevens that are misclassified as ones.
%
% <include>exp_mgr_metric_SevensAsOnes.m</include>
