%% Create a Deep Learning Experiment for Regression
% This example shows how to train a deep learning network for regression by
% using *<docid:nnet_ref#mw_11ce840f-eed7-4975-9c28-baeebd79f092 Experiment
% Manager>*. In this example, you use a regression model to predict the
% angles of rotation of handwritten digits. A custom metric function
% determines the fraction of angle predictions within an acceptable error
% margin from the true angles. For more information on using a regression
% model, see <docid:nnet_ug#bvlw7s_ Train Convolutional Neural Network for
% Regression>.

% Copyright 2019-2021 The MathWorks, Inc.

%% Open Experiment
% First, open the example. Experiment Manager loads a project with a
% preconfigured experiment that you can inspect and run. To open the
% experiment, in the *Experiment Browser* pane, double-click the name of
% the experiment (|RegressionExperiment|).
% 
% <<../exp_mgr_regression_config.png>>

% Project Name: 'DigitRegressionWithAccuracyProject'

%% 
% Built-in training experiments consist of a description, a table of
% hyperparameters, a setup function, and a collection of metric functions
% to evaluate the results of the experiment. For more information, see
% <docid:nnet_ref#mw_92d8f99c-f283-4bc5-a5d7-7d779c4831f7 Configure Deep
% Learning Experiment>.
%
% The *Description* field contains a textual description of the experiment.
% For this example, the description is:
%
%  Regression model to predict angles of rotation of digits, using hyperparameters to specify:
%  * the number of filters used by the convolution layers
%  * the probability of the dropout layer in the network
%
% The *Hyperparameters* section specifies the strategy (|Exhaustive Sweep|)
% and hyperparameter values to use for the experiment. When you run the
% experiment, Experiment Manager trains the network using every combination
% of hyperparameter values specified in the hyperparameter table. This
% example uses two hyperparameters:
%
% * |Probability| sets the probability of the dropout layer in the neural
% network. By default, the values for this hyperparameter are specified as
% |[0.1 0.2]|.
% * |Filters| indicates the number of filters used by the first convolution
% layer in the neural network. In the subsequent convolution layers, the
% number of filters is a multiple of this value. By default, the values of
% this hyperparameter are specified as |[4 6 8]|.
%
% The *Setup Function* configures the training data, network architecture,
% and training options for the experiment. The input to the setup function
% is a structure with fields from the hyperparameter table. The setup
% function returns four outputs that you use to train a network for image
% regression problems. The setup function has three sections.
% 
% * *Load Training Data* defines the training and validation data for the
% experiment as 4-D arrays. The training and validation data sets each
% contain 5000 images of digits from 0 to 9. The regression values
% correspond to the angles of rotation of the digits.
% 
%   [XTrain,~,YTrain] = digitTrain4DArrayData;
%   [XValidation,~,YValidation] = digitTest4DArrayData;
%
% * *Define Network Architecture* defines the architecture for a
% convolutional neural network for regression.
%
%   inputSize = [28 28 1];
%   numFilters = params.Filters;
%   layers = [
%       imageInputLayer(inputSize)
%       convolution2dLayer(3,numFilters,Padding="same")
%       batchNormalizationLayer
%       reluLayer
%       averagePooling2dLayer(2,Stride=2)
%       convolution2dLayer(3,2*numFilters,Padding="same")
%       batchNormalizationLayer
%       reluLayer
%       averagePooling2dLayer(2,Stride=2)
%       convolution2dLayer(3,4*numFilters,Padding="same")
%       batchNormalizationLayer
%       reluLayer
%       convolution2dLayer(3,4*numFilters,Padding="same")
%       batchNormalizationLayer
%       reluLayer
%       dropoutLayer(params.Probability)
%       fullyConnectedLayer(1)
%       regressionLayer];
%
% * *Specify Training Options* defines a <docid:nnet_ref#bu59f0q
% trainingOptions> object for the experiment. The example trains the
% network for 30 epochs. The learning rate is initially 0.001 and drops by
% a factor of 0.1 after 20 epochs. The software trains the network on the
% training data and calculates the root mean squared error (RMSE) and loss
% on the validation data at regular intervals during training. The
% validation data is not used to update the network weights.
% 
%   miniBatchSize = 128;
%   validationFrequency = floor(numel(YTrain)/miniBatchSize);
%   options = trainingOptions("sgdm", ...
%       MiniBatchSize=miniBatchSize, ...
%       MaxEpochs=30, ...
%       InitialLearnRate=1e-3, ...
%       LearnRateSchedule="piecewise", ...
%       LearnRateDropFactor=0.1, ...
%       LearnRateDropPeriod=20, ...
%       Shuffle="every-epoch", ...
%       ValidationData={XValidation,YValidation}, ...
%       ValidationFrequency=validationFrequency, ...
%       Verbose=false);
% 
% To inspect the setup function, under *Setup Function*, click *Edit*. The
% setup function opens in MATLAB(R) Editor. In addition, the code for the
% setup function appears in <#6 Appendix 1> at the end of this example.
% 
% The *Metrics* section specifies optional functions that evaluate the
% results of the experiment. Experiment Manager evaluates these functions
% each time it finishes training the network. To inspect a metric function,
% select the name of the metric function and click *Edit*. The metric
% function opens in MATLAB Editor.
%
% This example includes a metric function |Accuracy| that determines the
% percentage of angle predictions within an acceptable error margin from
% the true angles. By default, the function uses a threshold of 10 degrees.
% The code for the metric function appears in <#7 Appendix 2> at the end of
% this example.

%% Run Experiment
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function six times. Each trial uses a different
% combination of hyperparameter values. By default, Experiment Manager runs
% one trial at a time. If you have Parallel Computing Toolbox(TM), you can
% run multiple trials at the same time. For best results, before you run
% your experiment, start a parallel pool with as many workers as GPUs. For
% more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel> and
% <docid:distcomp_ug#mw_57e04559-0b60-42d5-ad55-e77ec5f5865f GPU Support by
% Release>.
%
% * To run one trial of the experiment at a time, on the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available.
%
% A table of results displays the RMSE and loss for each trial. The table
% also displays the accuracy of the trial, as determined by the custom
% metric function |Accuracy|.
% 
% <<../exp_mgr_regression_results.png>>
% 
% While the experiment is running, click *Training Plot* to display the
% training plot and track the progress of each trial.
% 
% <<../exp_mgr_regression_training.png>>

%% Evaluate Results
% To find the best result for your experiment, sort the table of results by
% accuracy. 
% 
% # Point to the *Accuracy* column. 
% # Click the triangle icon.
% # Select *Sort in Descending Order*.
%
% The trial with the highest accuracy appears at the top of the results
% table.
%
% <<../exp_mgr_regression_results_sorted.png>>
%
% To test the performance of an individual trial, export the trained
% network and display a box plot of the residuals for each digit class.
%
% # Select the trial with the highest accuracy.
% # On the *Experiment Manager* toolstrip, click *Export*.
% # In the dialog window, enter the name of a workspace variable for the
% exported network. The default name is |trainedNetwork|.
% # Use the exported network as the input to the function |plotResiduals|,
% which is listed in <#8 Appendix 3> at the end of this example. For
% instance, in the MATLAB Command Window, enter:
%
%   plotResiduals(trainedNetwork)
%
% The function creates a residual box plot for each digit. The digit
% classes with highest accuracy have a mean close to zero and little
% variance.
%
% <<../exp_mgr_regression_residual.png>>
%
% To record observations about the results of your experiment, add an
% annotation. 
%
% # In the results table, right-click the *Accuracy* cell of the best
% trial.
% # Select *Add Annotation*. 
% # In the *Annotations* pane, enter your observations in the text box.
%
% <<../exp_mgr_regression_annotation.png>>
%
% For more information, see
% <docid:nnet_ref#mw_ffdb175d-cd27-4954-8fa6-e006dbb6dfd0 Sort, Filter, and
% Annotate Experiment Results>.

%% Close Experiment
% In the *Experiment Browser* pane, right-click the name of the project and
% select *Close Project*. Experiment Manager closes all of the experiments
% and results contained in the project.

%% Appendix 1: Setup Function
% This function configures the training data, network architecture, and
% training options for the experiment.
% 
% *Input*
%
% * |params| is a structure with fields from the Experiment Manager
% hyperparameter table.
%
% *Output*
%
% * |XTrain| is a 4-D array containing the training data.
% * |YTrain| is a 1-D array containing the regression values for training,
% * |layers| is a layer graph that defines the neural network architecture.
% * |options| is a |trainingOptions| object.
%
% <include>exp_mgr_regression_setup_function.m</include>

%% Appendix 2: Compute Accuracy of Regression Model
% This function calculates the number of predictions within an acceptable
% error margin from the true angles.
%
% <include>exp_mgr_regression_Accuracy.m</include>

%% Appendix 3: Display Box Plot of Residuals for Each Digit
% This function creates a residual box plot for each digit.
%
% <include>exp_mgr_regression_plotResiduals.m</include>