%% Sweep Hyperparameters to Train a Classification Network
% This example shows how to use the default experiment setup function to
% train an image classification network by sweeping hyperparameters. For
% more examples of solving image classification problems with Experiment
% Manager, see <docid:nnet_ug#mw_3df568f0-f5d4-4de9-b443-dff6c3dad088
% Create a Deep Learning Experiment for Classification> and
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel>. For more information on an
% alternative strategy to sweeping hyperparameters, see
% <docid:nnet_ug#mw_b7632999-289d-4df6-88b6-64532a315bcb Tune Experiment
% Hyperparameters by Using Bayesian Optimization>.

% Copyright 2019-2020 The MathWorks, Inc.

%%
% Open the example to load a project with a preconfigured experiment that
% you can inspect and run. To open the experiment, in the *Experiment
% Browser* pane, double-click the name of the experiment (|Experiment1|).
%
% <<../app_exp_mgr_classification_config.png>>

% Project Name: 'DigitClassificationProject'

%%
% Alternatively, you can configure the experiment yourself by following
% these steps.
%
% 1. Open Experiment Manager.
%
% 2. Click *New* > *Project* and select the location and name for a new
% project. Experiment Manager opens a new experiment in the project. The
% *Experiment* pane displays the description, hyperparameters, setup
% function, and metrics that define the experiment.
%
% 3. In the *Description* box, enter a description of the experiment:
%
%  Classification of digits, using various initial learning rates.
%
% 4. Under *Hyperparameters*, replace the value of |myInitialLearnRate|
% with |0.0025:0.0025:0.015|. Verify that *Strategy* is set to |Exhaustive
% Sweep|.
%
% 5. Under *Setup Function*, click *Edit*. The setup function opens in
% MATLAB Editor. The setup function specifies the training data, network
% architecture, and training options for the experiment. By default, the
% template for the setup function has three sections.
% 
% * *Load Image Data* defines image datastores containing the training and
% validation data for the experiment. The experiment uses the Digits data
% set, which consists of 10,000 28-by-28 pixel grayscale images of digits
% from 0 to 9, categorized by the digit they represent. For more
% information on this data set, see
% <docid:nnet_ug#mw_d3130147-29e6-47eb-b6f3-9a11a4c8b659 Image Data Sets>.
% * *Define Network Architecture* defines the architecture for a simple
% convolutional neural network for deep learning classification.
% * *Specify Training Options* defines a |<docid:nnet_ref#bu59f0q
% trainingOptions>| object for the experiment. By default, the template
% loads the values for the training option |'InitialLearnRate'| from the
% |myInitialLearnRate| entry in the hyperparameter table.

%%
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function six times. Each trial uses one of the
% learning rates specified in the hyperparameter table. By default,
% Experiment Manager runs one trial at a time. If you have Parallel
% Computing Toolbox, you can run multiple trials at the same time. For best
% results, before you run your experiment, start a parallel pool with as
% many workers as GPUs. For more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel>.
%
% * To run one trial of the experiment at a time, in the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available.
%
% A table of results displays the accuracy and loss for each trial.
%
% <<../app_exp_mgr_classification_results.png>>
% 
% While the experiment is running, click *Training Plot* to display the
% training plot and track the progress of each trial. You can also monitor
% the training progress in the MATLAB Command Window.
%
% <<../app_exp_mgr_classification_training.png>>
%
% Click *Confusion Matrix* to display the confusion matrix for the
% validation data in each completed trial.
%
% <<../app_exp_mgr_classification_confusion.png>>
%
% When the experiment finishes, you can sort the table by column or filter
% trials by using the *Filters* pane. For more information, see
% <docid:nnet_ref#mw_ffdb175d-cd27-4954-8fa6-e006dbb6dfd0 Sort and Filter
% Experiment Results>.
%
% To test the performance of an individual trial, export the trained
% network or the training information for the trial. On the *Experiment
% Manager* toolstrip, select *Export* > *Trained Network* or *Export* >
% *Training Information*, respectively. For more information, see
% <docid:nnet_ref#bu6sn4c-trainedNet Trained Network> and
% <docid:nnet_ref#bu6sn4c-traininfo Training Information>.

%%
% To close the experiment, in the *Experiment Browser* pane, right-click
% the name of the project and select *Close Project*. Experiment Manager
% closes all of the experiments and results contained in the project.