%% Classify Images on FPGA by Using Quantized GoogLeNet Network
% This example show how to use the Deep Learning HDL Toolbox™ to deploy a quantized GoogleNet network to classify an image. The example uses the pretrained GoogLeNet network to demonstrate transfer learning, quantization, and deployment for the quantized network. Quantization helps reduce the memory requirement of a deep neural network by quantizing weights, biases and activations of network layers to 8-bit scaled integer data types. Use MATLAB® to retrieve the prediction results.
% Deploy the quantized GoogLeNet network by creating a dlhdl.Workflow object. Use the dlhdl.Workflow object to:
% Generate a list of instructions, weights and biases by using the compile method.
% Generate a programming file for the FPGA by using the deploy method.
% Retrieve the network prediction results and performance by using the predict method.
% GoogLeNet has been trained on over a million images and can classify images into 1000 object categories (such as keyboard, coffee mug, pencil, and many animals). The network has learned rich feature representations for a wide range of images. The network takes an image as input, and then outputs a label for the object in the image together with the probabilities for each of the object categories.
%% Prerequisites
% Deep Learning Toolbox™
% Deep Learning HDL Toolbox™
% Deep Learning Toolbox Model for GoogLeNet Network
% Deep Learning HDL Toolbox™ Support Package for Intel FPGA and SoC
% Image Processing Toolbox™
% Intel Arria10 SoC development kit
% Deep Learning Toolbox™ Model Quantization Library support package.
% MATLAB Coder Interface for Deep learning Libraries
%
%% Transfer Learning Using GoogLeNet
% To perform classification on a new set of images, you fine-tune a pretrained GoogLeNet convolutional neural network by transfer learning. In transfer learning, you can take a pretrained network and use it as a starting point to learn a new task. Fine-tuning a network with transfer learning is usually much faster and easier than training a network with randomly initialized weights from scratch. You can quickly transfer learned features to a new task using a smaller number of training images.
%
%% Load Pretrained DAG Network
% Load the pretrained DAG network, GoogLeNet.

net = googlenet;

% Use the analyzeNetwork function to obtain information about the network layers.

analyzeNetwork(net);

% The first layer, the image input layer, requires input images of size 224-by-224-by-3, where 3 is the number of color channels.

inputSize = net.Layers(1).InputSize

%
%% Define Training and Validation Data Sets
% This example uses the MathWorks MerchData data set. This is a small data set containing 75 images of MathWorks merchandise, belonging to five different classes (cap, cube, playing cards, screwdriver, and torch).

unzip('MerchData.zip');
imds = imageDatastore('MerchData', ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');

% Divide the data into training and validation data sets. Use 70% of the images for training and 30% for validation. splitEachLabel splits the images datastore into two new datastores.

[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7,'randomized');

% This data set now contains 55 training images and 20 validation images. Display some sample images.

numTrainImages = numel(imdsTrain.Labels);
idx = randperm(numTrainImages,16);
figure
for i = 1:16
    subplot(4,4,i)
    I = readimage(imdsTrain,idx(i));
    imshow(I)
end

%
%% Replace Final Layers
% The fully connected layer and classification layer of the pretrained network net are configured for 1000 classes. These two layers, loss3-classifier and output in GoogLeNet, contain information on how to combine the features that the network extracts into class probabilities, a loss value, and predicted labels. To retrain a pretrained network to classify new images, replace these two layers with new layers adapted to the new data set.
% Extract the layer graph from the trained network.

lgraph = layerGraph(net)

% Replace the fully connected layer with a new fully connected layer that has number of outputs equal to the number of classes. To make learning faster in the new layers than in the transferred layers, increase the WeightLearnRateFactor and BiasLearnRateFactor values of the fully connected layer.

numClasses = numel(categories(imdsTrain.Labels))

% Remove 'loss3-classifier', 'prob' and 'output' layers from the lgraph.

layers = net.SortedLayers;
for i = 0:2
    lgraph = removeLayers(lgraph,layers(end-i).Name);
end

% Create three new layers and add them to the lgraph. Ensure the transferred and new layers are properly connected together in the lgraph.

newLayers = [
    fullyConnectedLayer(numClasses,'WeightLearnRateFactor',20,'BiasLearnRateFactor',20,'Name','newFC')
    softmaxLayer('Name','newProb')
    classificationLayer('Name','newClassOutput',"Classes","auto")];

lgraph = addLayers(lgraph,newLayers);
lgraph = connectLayers(lgraph,layers(end-3).Name,'newFC');

%% Train Network
% The network requires input images of size 224-by-224-by-3, but the images in the image datastores have different sizes. Use an augmented image datastore to automatically resize the training images. Specify additional augmentation operations to perform on the training images: randomly flip the training images along the vertical axis, and randomly translate them up to 30 pixels horizontally and vertically. Data augmentation helps prevent the network from over-fitting and memorizing the exact details of the training images.

pixelRange = [-30 30];
imageAugmenter = imageDataAugmenter( ...
    'RandXReflection',true, ...
    'RandXTranslation',pixelRange, ...
    'RandYTranslation',pixelRange);
augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain, ...
    'DataAugmentation',imageAugmenter);

% To automatically resize the validation images without performing further data augmentation, use an augmented image datastore without specifying any additional preprocessing operations.

augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation);

% Specify the training options. For transfer learning, keep the features from the early layers of the pretrained network (the transferred layer weights). To slow down learning in the transferred layers, set the initial learning rate to a small value. In the previous step, the learning rate factors were increased for the fully conencted layer to speed up learning in the new final layers. This combination of learning rate settings results in fast learning only in the new layers and slower learning in the other layers. When performing transfer learning, you do not need to train for as many epochs. An epoch is a full training cycle on the entire training data set. Specify the mini-batch size to be 11. The software validates the network every ValidationFrequency iterations during training.

options = trainingOptions('sgdm', ...
    'MiniBatchSize',11, ...
    'MaxEpochs',5, ...
    'InitialLearnRate',2e-4, ...
    'Shuffle','every-epoch', ...
    'ValidationData',augimdsValidation, ...
    'ValidationFrequency',3, ...
    'Verbose',false, ...
    'Plots','training-progress');

% Train the network that consists of the transferred and new layers. By default, trainNetwork uses a GPU if one is available (requires Parallel Computing Toolbox™ and a supported GPU device. For more information, see GPU Support by Release (Parallel Computing Toolbox)). Otherwise, the network uses a CPU (requires MATLAB Coder Interface for Deep learning Libraries™). You can also specify the execution environment by using the 'ExecutionEnvironment' name-value argument of trainingOptions.

netTransfer = trainNetwork(augimdsTrain,lgraph,options);

%% Create dlquantizer Object
% Create a quantized network by using the dlquantizer object. Set the target execution environment to FPGA..

dlQuantObj = dlquantizer(netTransfer,'ExecutionEnvironment','FPGA');

%% Calibrate Quantized Network
% Use the calibrate function to exercise the network by using sample inputs to collect the range information. The calibrate function exercises the network and collects the dynamic ranges for the learnable parameters of the convolution and fully connected layers of the network.
% For best quantization results, the calibration data must be a representative of actual inputs that are predicted by the network.

dlQuantObj.calibrate(augimdsTrain);

%% Set Up Intel Quartus Prime Standard
% Set the synthesis tool path to point to an installed Intel® Quartus® Prime Standard Edition 18.1 executable file. You must have already installed Altera® Quartus II.

% hdlsetuptoolpath('ToolName','Altera Quartus II','ToolPath','C:\intel\18.1\quartus\bin\quartus.exe');

%% Create Target Object
% Create a target object with a custom name for your target device and an interface to connect your target device to the host computer. Interface options are JTAG and Ethernet.

hTarget = dlhdl.Target('Intel','Interface','JTAG');

% Alternatively, you can use an Ethernet interface.

% hTarget = dlhdl.Target('Intel','Interface','Ethernet');

%% Create Workflow Object
% Create an object of the dlhdl.Workflow class. Specify dlQuantObj as the network. Make sure that the bitstream name matches the data type and the FPGA board that you are targeting. In this example, the target FPGA board is the Intel Arria10 SOC board and the bitstream uses the int8 data type.

hW = dlhdl.Workflow('network', dlQuantObj, 'Bitstream', 'arria10soc_int8','Target',hTarget);

%% Compile Workflow Object
% To compile the GoogLeNet network, run the compile function of the dlhdl.Workflow object.

dn = hW.compile

%% Program Bitstream onto FPGA and Download Network Weights
% To deploy the network on the Intel Arria10 SoC hardware, run the deploy function of the dlhdl.Workflow object . This function uses the output of the compile function to program the FPGA board by using the programming file. The function also downloads the network weights and biases. The deploy function starts programming the FPGA device, displays progress messages, and the time it takes to deploy the network.

hW.deploy

%% Load Example Image

I = imresize(readimage(imdsValidation,1),[224 224]);
figure
imshow(I)

%% Retrieve Image Prediction
% Execute the predict function of the dlhdl.Workflow object and display the prediction results.

[prediction, speed] = hW.predict(single(I),'Profile','off');
[val, index] = max(prediction);
label = netTransfer.Layers(end).ClassNames{index}
title(string(label));

%% Retrieve Deployed Network Performance
% View the performance ofd the deployed network by using the predict method with the Profile argument set to on.

[~, speed] = hW.predict(single(I),'Profile','on')

% The speed table contains the latency information for every layer, total network latency, and the overall network performance in frames per second (FPS). For more information, see Profile Inference Run.
%
%
% Copyright 2020 The MathWorks, Inc.