%% Create a Deep Learning Experiment for Classification
% This example shows how to train a deep learning network for
% classification by using
% *<docid:nnet_ref#mw_11ce840f-eed7-4975-9c28-baeebd79f092 Experiment
% Manager>*. In this example, you train two networks to classify images of
% MathWorks merchandise into five classes. Each network is trained using
% three algorithms. In each case, a confusion matrix compares the true
% classes for a set of validation images with the classes predicted by the
% trained network. For more information on training a network for image
% classification, see
% <docid:nnet_ug#mw_50469739-1a06-4865-b1b2-e9c149b86f96 Train Deep
% Learning Network to Classify New Images>.

% Copyright 2019-2020 The MathWorks, Inc.

%% Open Experiment
% First, open the example. Experiment Manager loads a project with a
% preconfigured experiment that you can inspect and run. To open the
% experiment, in the *Experiment Browser* pane, double-click the name of
% the experiment (|ClassificationExperiment|).
% 
% <<../exp_mgr_classification_config.png>>

% Project Name: 'MerchandiseClassificationProject'

%% 
% An experiment definition consists of a description, a table of
% hyperparameters, a setup function, and (optionally) a collection of
% metric functions to evaluate the results of the experiment. For more
% information, see <docid:nnet_ref#mw_92d8f99c-f283-4bc5-a5d7-7d779c4831f7
% Configure Deep Learning Experiment>.
%
% The *Description* box contains a textual description of the experiment.
% For this example, the description is:
%
%  Merchandise image classification using:
%  - an untrained network (default) or a pretrained network (googlenet)
%  - various solvers for training networks (sgdm, rmsprop, or adam)
%
% The *Hyperparameters* section specifies the strategy (|Exhaustive Sweep|)
% and hyperparameter values to use for the experiment. When you run the
% experiment, Experiment Manager trains the network using every combination
% of hyperparameter values specified in the hyperparameter table. This
% example uses two hyperparameters:
%
% * |Network| specifies the network to train. The options include
% |"default"| (the default classification network provided by the setup
% function template) and |"googlenet"| (a pretrained GoogLeNet network with
% modified layers for transfer learning).
% * |Solver| indicates the algorithm used to train the network. The options
% include |"sgdm"| (stochastic gradient descent with momentum), |"rmsprop"|
% (root mean square propagation), and |"adam"| (adaptive moment
% estimation). For more information about these algorithms, see
% <docid:nnet_ref#bu80qkw-3>.
%
% The *Setup Function* configures the training data, network architecture,
% and training options for the experiment. To inspect the setup function,
% under *Setup Function*, click *Edit*. The setup function opens in
% MATLAB(R) Editor.
% 
% In this example, the input to the setup function is a |struct| with
% fields from the hyperparameter table. The setup function returns three
% outputs that you use to train a network for image classification
% problems. The setup function has three sections.
% 
% * *Load Image Data* defines image datastores containing the training and
% validation data. This example loads images from the file |MerchData.zip|.
% This small data set contains 75 images of MathWorks merchandise,
% belonging to five different classes. The images are of size
% 227-by-227-by-3. For more information on this data set, see
% <docid:nnet_ug#mw_d3130147-29e6-47eb-b6f3-9a11a4c8b659 Image Data Sets>.
% * *Define Network Architecture* defines the architecture for a
% convolutional neural network for deep learning classification. In this
% example, the choice of network to train depends on the value of the
% hyperparameter |Network|.
% * *Specify Training Options* defines a |<docid:nnet_ref#bu59f0q
% trainingOptions>| object for the experiment. The example trains the
% network for 10 epochs using the algorithm specified by the |Solver| entry
% in the hyperparameter table.
%
% The *Metrics* section specifies optional functions that evaluate the
% results of the experiment. This example does not include any custom
% metric functions.

%% Run Experiment
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function six times. Each trial uses a different
% combination of hyperparameter values. By default, Experiment Manager runs
% one trial at a time. If you have Parallel Computing Toolbox(TM), you can
% run multiple trials at the same time. For best results, before you run
% your experiment, start a parallel pool with as many workers as GPUs. For
% more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel>.
%
% * To run one trial of the experiment at a time, in the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available.
%
% A table of results displays the accuracy and loss for each trial.
% 
% <<../exp_mgr_classification_results.png>>
% 
% While the experiment is running, click *Training Plot* to display the
% training plot and track the progress of each trial.
% 
% <<../exp_mgr_classification_training.png>>

%% Evaluate Results
% To find the best result for your experiment, sort the table of results by
% validation accuracy. 
% 
% # Point to the *Validation Accuracy* column. 
% # Click the triangle icon.
% # Select *Sort in Descending Order*.
%
% The trial with the highest validation accuracy appears at the top of the
% results table.
% 
% <<../exp_mgr_classification_results_sorted.png>>
% 
% To display the confusion matrix for this trial, select the top row in the
% results table and click *Confusion Matrix*.
% 
% <<../exp_mgr_classification_confusion.png>> 


%% Close Experiment
% In the *Experiment Browser* pane, right-click the name of the project and
% select *Close Project*. Experiment Manager closes all of the experiments
% and results contained in the project.