%% Experiment with Weight Initializers for Transfer Learning
% This example shows how to configure an experiment that initializes the
% weights of convolution and fully connected layers using different weight
% initializers for training. To compare the performance of different weight
% initializers for your task, create an experiment using this example as a
% guide.
%
% When training a deep learning network, the initialization of layer
% weights and biases can have a big impact on how well the network trains.
% The choice of initializer has a bigger impact on networks without batch
% normalization layers. For more information on weight initializers, see
% <docid:nnet_ug#mw_36834ae2-c984-4c57-9cb9-acec6756c335 Compare Layer
% Weight Initializers>.

% Copyright 2019-2021 The MathWorks, Inc.

%% Open Experiment
% First, open the example.
% *<docid:nnet_ref#mw_11ce840f-eed7-4975-9c28-baeebd79f092 Experiment
% Manager>* loads a project with a preconfigured experiment that you can
% inspect and run. To open the experiment, in the *Experiment Browser*
% pane, double-click the name of the experiment
% (|WeightInitializerExperiment|).
%
% <<../exp_mgr_weight_init_config.png>>

% Project Name: 'FlowerWeightInitializerProject'

%%
% Built-in training experiments consist of a description, a table of
% hyperparameters, a setup function, and a collection of metric functions
% to evaluate the results of the experiment. For more information, see
% <docid:nnet_ref#mw_92d8f99c-f283-4bc5-a5d7-7d779c4831f7 Configure Deep
% Learning Experiment>.
%
% The *Description* field contains a textual description of the experiment.
% For this example, the description is:
%
%  Perform transfer learning by initializing the weights of convolution and
%  fully connected layers in a pretrained network.
%
% The *Hyperparameters* section specifies the strategy (|Exhaustive Sweep|)
% and hyperparameter values to use for the experiment. When you run the
% experiment, Experiment Manager trains the network using every combination
% of hyperparameter values specified in the hyperparameter table. This
% example uses the hyperparameters |WeightsInitializer| and
% |BiasInitializer| to specify the weight and bias initializers for the
% convolution and fully connected layers in a pretrained network. For more
% information about these initializers, see
% |<docid:nnet_ref#mw_2d97b6cd-f8aa-4fad-88d6-d34875484820.mw_bec5cf10-a8e8-4560-be5a-c4ccb6594b02
% WeightsInitializer>| and
% |<docid:nnet_ref#mw_2d97b6cd-f8aa-4fad-88d6-d34875484820.mw_db0ca0f4-7628-453c-b24a-33c73d5c065e
% BiasInitializer>|.
%
% The *Setup Function* configures the training data, network architecture,
% and training options for the experiment. The input to the setup function
% is a structure with fields from the hyperparameter table. The setup
% function returns three outputs that you use to train a network for image
% classification problems. In this example, the setup function:
%
% * Loads a pretrained GoogLeNet network.
%
%   lgraph = googlenet(Weights="none");
%
% * Downloads and extracts the Flowers data set, which is about 218 MB. For
% more information on this data set, see
% <docid:nnet_ug#mw_d3130147-29e6-47eb-b6f3-9a11a4c8b659 Image Data Sets>.
%   
%   url = "http://download.tensorflow.org/example_images/flower_photos.tgz";
%   downloadFolder = tempdir;
%   filename = fullfile(downloadFolder,"flower_dataset.tgz");
%   
%   imageFolder = fullfile(downloadFolder,"flower_photos");
%   if ~exist(imageFolder,"dir")
%       disp("Downloading Flower Dataset (218 MB)...")
%       websave(filename,url);
%       untar(filename,downloadFolder)
%   end
%   
%   imds = imageDatastore(imageFolder, ...
%       IncludeSubfolders=true, ...
%       LabelSource="foldernames");
%   
%   [imdsTrain,imdsValidation] = splitEachLabel(imds,0.9);
%   inputSize = net.Layers(1).InputSize;
%   augimdsTrain = augmentedImageDatastore(inputSize,imdsTrain);
%   augimdsValidation = augmentedImageDatastore(inputSize,imdsValidation);
%
% * Initializes the input weight in the convolution and fully connected
% layers by using the initializers specified in the hyperparameter table.
% The auxiliary function |findLayersToReplace|, which is listed in <#6
% Appendix 2> at the end of this example, determines the layers in the
% network architecture that can be modified for transfer learning.
%   
%   numClasses = numel(categories(imdsTrain.Labels));
%   weightsInitializer = params.WeightsInitializer;
%   biasInitializer = params.BiasInitializer;
%   
%   learnableLayer = findLayersToReplace(lgraph);
%   newLearnableLayer = fullyConnectedLayer(numClasses,Name="new_fc");
%   lgraph = replaceLayer(lgraph,learnableLayer.Name,newLearnableLayer);
%   
%   for i = 1:numel(lgraph.Layers)
%       layer = lgraph.Layers(i);
%       
%       if class(layer) == "nnet.cnn.layer.Convolution2DLayer" || ...
%               class(layer) == "nnet.cnn.layer.FullyConnectedLayer"
%           layerName = layer.Name;
%           newLayer = layer;
%           
%           newLayer.WeightsInitializer = weightsInitializer;
%           newLayer.BiasInitializer = biasInitializer;
%           
%           lgraph = replaceLayer(lgraph,layerName,newLayer);
%       end
%   end
%   
% * Defines a <docid:nnet_ref#bu59f0q trainingOptions> object for the
% experiment. The example trains the network for 10 epochs, using a
% mini-batch size of 128 and validating the network every 5 epochs.
%
%   miniBatchSize = 128;
%   validationFrequencyEpochs = 5;
%   
%   numObservations = augimdsTrain.NumObservations;
%   numIterationsPerEpoch = floor(numObservations/miniBatchSize);
%   validationFrequency = validationFrequencyEpochs * numIterationsPerEpoch;
%   
%   options = trainingOptions("sgdm", ...
%       MaxEpochs=10, ...
%       MiniBatchSize=miniBatchSize, ...
%       Shuffle="every-epoch", ...
%       ValidationData=augimdsValidation, ...
%       ValidationFrequency=validationFrequency, ...
%       Verbose=false);
% 
% To inspect the setup function, under *Setup Function*, click *Edit*. The
% setup function opens in MATLAB(R) Editor. In addition, the code for the
% setup function appears in <#5 Appendix 1> at the end of this example.
%
% The *Metrics* section specifies optional functions that evaluate the
% results of the experiment. This example does not include any custom
% metric functions.

%% Run Experiment
% When you run the experiment, Experiment Manager trains the network
% defined by the setup function multiple times. Each trial uses a different
% combination of hyperparameter values. By default, Experiment Manager runs
% one trial at a time. If you have Parallel Computing Toolbox(TM), you can
% run multiple trials at the same time. For best results, before you run
% your experiment, start a parallel pool with as many workers as GPUs. For
% more information, see
% <docid:nnet_ug#mw_4858679a-7447-4736-9c57-7b8dc706648d Use Experiment
% Manager to Train Networks in Parallel> and
% <docid:distcomp_ug#mw_57e04559-0b60-42d5-ad55-e77ec5f5865f GPU Support by
% Release>.
%
% * To run one trial of the experiment at a time, on the Experiment Manager
% toolstrip, click *Run*.
% * To run multiple trials at the same time, click *Use Parallel* and then
% *Run*. If there is no current parallel pool, Experiment Manager starts
% one using the default cluster profile. Experiment Manager then executes
% multiple simultaneous trials, depending on the number of parallel workers
% available.
%
% A table of results displays the accuracy and loss for each trial. While
% the experiment is running, click *Training Plot* to display the training
% plot and track the progress of each trial. Click *Confusion Matrix* to
% display the confusion matrix for the validation data in each completed
% trial.
%
% <<../exp_mgr_weight_init_results.png>>
%
% Note that, for the trials that use the He weight initializer, Experiment
% Manager interrupts the training because the training and validation loss
% become undefined after a few iterations. Continuing the training for
% those trials does not produce any useful results.
%
% When the experiment finishes, you can sort the results table by column,
% filter trials by using the *Filters* pane, or record observations by
% adding annotations. For more information, see
% <docid:nnet_ref#mw_ffdb175d-cd27-4954-8fa6-e006dbb6dfd0 Sort, Filter, and
% Annotate Experiment Results>.
%
% To test the performance of an individual trial, export the trained
% network or the training information for the trial. On the *Experiment
% Manager* toolstrip, select *Export* > *Trained Network* or *Export* >
% *Training Information*, respectively. For more information, see
% <docid:nnet_ref#bu6sn4c-trainedNet Trained Network> and
% <docid:nnet_ref#bu6sn4c-traininfo Training Information>.

%% Close Experiment
% In the *Experiment Browser* pane, right-click the name of the project and
% select *Close Project*. Experiment Manager closes all of the experiments
% and results contained in the project.

%% Appendix 1: Setup Function
% This function configures the training data, network architecture, and
% training options for the experiment.
%
% *Input*
%
% * |params| is a structure with fields from the Experiment Manager
% hyperparameter table.
%
% *Output*
%
% * |augimdsTrain| is an augmented image datastore object for the training
% data.
% * |lgraph| is a layer graph that defines the neural network architecture.
% * |options| is a |trainingOptions| object.
%
% <include>exp_mgr_weight_init_setup_function.m</include>

%% Appendix 2: Find Layers to Replace
% This function finds the single classification layer and the
% preceding learnable (fully connected or convolutional) layer of the layer
% graph |lgraph|.
%
% <include>exp_mgr_findLayersToReplace.m</include>