%% Sweep Hyperparameters to Train Regression Network
% This example shows how to configure an experiment to train an image
% regression network by sweeping hyperparameters. For another example of
% solving a regression problem with Experiment Manager, see
% <docid:nnet_ug#mw_b8fa2d47-903b-4f8e-8c69-8eedbb538b15 Create a Deep
% Learning Experiment for Regression>.

% Copyright 2019-2020 The MathWorks, Inc.

%%
% Open the example to load a project with a preconfigured experiment that
% you can inspect and run. To open the experiment, in the *Experiment
% Browser* pane, double-click the name of the experiment (|Experiment1|).
%
% <<../app_exp_mgr_regression_config.png>>

% Project Name: 'DigitRegressionProject'

%%
% Alternatively, you can configure the experiment yourself by following
% these steps.
%
% 1. Open Experiment Manager.
%
% 2. Click *New* > *Project* and select the location and name for a new
% project. Experiment Manager opens a new experiment in the project. The
% *Experiment* pane displays the description, hyperparameters, setup
% function, and metrics that define the experiment.
%
% 3. In the *Description* box, enter a description of the experiment:
%
%  Regression to predict angles of rotation of digits, using various initial learning rates.
%
% 4. Under *Hyperparameters*, replace the value of |myInitialLearnRate|
% with |0.001:0.001:0.006|. Verify that *Strategy* is set to |Exhaustive
% Sweep|.
%
% 5. Under *Setup Function*, click *Edit*. The setup function opens in
% MATLAB Editor.
% 
% * Modify the setup function signature to return four outputs. These
% outputs are used to call the |<docid:nnet_ref#bu6sn4c trainNetwork>|
% function to train a network for image regression problems.
%
%   function [XTrain,YTrain,layers,options] = Experiment1_setup1(params)
%
% * Modify the *Load Image Data* section of the setup function to define
% the training and validation data for the experiment as 4-D arrays. In
% this experiment, the training and validation data each consist of 5000
% images from the Digits data set. Each image shows a digit from 0 to 9,
% rotated by a certain angle. The regression values correspond to the
% angles of rotation. For more information on this data set, see
% <docid:nnet_ug#mw_d3130147-29e6-47eb-b6f3-9a11a4c8b659 Image Data Sets>.
% Be sure to delete all of the existing code in this section of the setup
% function.
%
%   [XTrain,~,YTrain] = digitTrain4DArrayData;
%   [XValidation,~,YValidation] = digitTest4DArrayData;
%
% * Modify the *Define Network Architecture* section of the setup function
% to define a convolutional neural network for regression. Be sure to
% delete all of the existing code in this section of the setup function.
%
%   layers = [
%       imageInputLayer([28 28 1])
%       convolution2dLayer(3,8,'Padding','same')
%       batchNormalizationLayer
%       reluLayer
%       averagePooling2dLayer(2,'Stride',2)
%       convolution2dLayer(3,16,'Padding','same')
%       batchNormalizationLayer
%       reluLayer
%       averagePooling2dLayer(2,'Stride',2)
%       convolution2dLayer(3,32,'Padding','same')
%       batchNormalizationLayer
%       reluLayer
%       convolution2dLayer(3,32,'Padding','same')
%       batchNormalizationLayer
%       reluLayer
%       dropoutLayer(0.2)
%       fullyConnectedLayer(1)
%       regressionLayer];
%
% * Modify the *Specify Training Options* section of the setup function to
% use the validation data in the 4-D arrays |XValidation| and
% |YValidation|. This section of the setup function loads the values for
% the training option |'InitialLearnRate'| from the |myInitialLearnRate|
% entry in the hyperparameter table.
%
%   options = trainingOptions('sgdm', ...
%       'MaxEpochs',5, ... 
%       'ValidationData',{XValidation,YValidation}, ...
%       'ValidationFrequency',30, ...
%       'InitialLearnRate',params.myInitialLearnRate);

%%
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function six times. Each trial uses one of the
% learning rates specified in the hyperparameter table. Each trial uses one
% of the learning rates specified in the hyperparameter table. By default,
% Experiment Manager runs one trial at a time. If you have Parallel
% Computing Toolbox, you can run multiple trials at the same time. For best
% results, before you run your experiment, start a parallel pool with as
% many workers as GPUs. For more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel>.
%
% * To run one trial of the experiment at a time, in the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available. 
%
% A table of results displays the root mean squared error (RMSE) and loss
% for each trial.
%
% <<../app_exp_mgr_regression_results.png>>
%
% While the experiment is running, click *Training Plot* to display the
% training plot and track the progress of each trial. You can also monitor
% the training progress in the MATLAB Command Window.
%
% <<../app_exp_mgr_regression_training.png>>
%
% When the experiment finishes, you can sort the table by column or filter
% trials by using the *Filters* pane. For more information, see
% <docid:nnet_ref#mw_ffdb175d-cd27-4954-8fa6-e006dbb6dfd0 Sort and Filter
% Experiment Results>.
% 
% To test the performance of an individual trial, export the trained
% network or the training information for the trial. On the *Experiment
% Manager* tab, select *Export* > *Trained Network* or *Export* > *Training
% Information*, respectively. For more information, see
% <docid:nnet_ref#bu6sn4c-trainedNet Trained Network> and
% <docid:nnet_ref#bu6sn4c-traininfo Training Information>.

%%
% To close the experiment, in the *Experiment Browser* pane, right-click
% the name of the project and select *Close Project*. Experiment Manager
% closes all of the experiments and results contained in the project.