%% Classify Images on an FPGA Using a Quantized DAG Network
% In this example, you use Deep Learning HDL Toolbox™ to deploy a quantized deep convolutional neural network and classify an image. The example uses the pretrained ResNet-18 convolutional neural network to demonstrate transfer learning, quantization, and deployment for the quantized network. Use MATLAB ® to retrieve the prediction results.
% ResNet-18 has been trained on over a million images and can classify images into 1000 object categories (such as keyboard, coffee mug, pencil, and many animals). The network has learned rich feature representations for a wide range of images. The network takes an image as input and outputs a label for the object in the image together with the probabilities for each of the object categories.
%% Required Products
% For this example, you need:
% Deep Learning Toolbox™
% Deep Learning HDL Toolbox™
% Deep Learning Toolbox Model for ResNet-18 Network
% Deep Learning HDL Toolbox™ Support Package for Xilinx® FPGA and SoC Devices
% Image Processing Toolbox™
% Deep Learning Toolbox Model Quantization Library
% MATLAB® Coder™ Interface for Deep Learning Libraries
%
%
%% Transfer Learning Using Resnet-18
% To perform classification on a new set of images, you fine-tune a pretrained ResNet-18 convolutional neural network by transfer learning. In transfer learning, you can take a pretrained network and use it as a starting point to learn a new task. Fine-tuning a network with transfer learning is usually much faster and easier than training a network with randomly initialized weights from scratch. You can quickly transfer learned features to a new task using a smaller number of training images.
%% Load Pretrained SeriesNetwork
% To load the pretrained network ResNet-18, enter:

snet = resnet18;

% To view the layers of the pretrained network, enter:

analyzeNetwork(snet);

% The first layer, the image input layer, requires input images of size 227-by-227-by-3, where 3 is the number of color channels.

inputSize = snet.Layers(1).InputSize;

%
%% Define Training and Validation Data Sets
% This example uses the MathWorks MerchData data set. This is a small data set containing 75 images of MathWorks merchandise, belonging to five different classes (cap, cube, playing cards, screwdriver, and torch).

curDir = pwd;
unzip('MerchData.zip');
imds = imageDatastore('MerchData', ...
'IncludeSubfolders',true, ...
'LabelSource','foldernames');
[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7,'randomized');

%
%% Replace Final Layers
% The fully connected layer and classification layer of the pretrained network net are configured for 1000 classes. These two layers fc1000 and  ClassificationLayer_predictions in ResNet-18, contain information on how to combine the features that the network extracts into class probabilities and predicted labels . These two layers must be fine-tuned  for the new classification problem. Extract all the layers, except the last  two layers, from the pretrained network.

lgraph = layerGraph(snet)
numClasses = numel(categories(imdsTrain.Labels))
newLearnableLayer = fullyConnectedLayer(numClasses, ...
'Name','new_fc', ...
'WeightLearnRateFactor',10, ...
'BiasLearnRateFactor',10);
lgraph = replaceLayer(lgraph,'fc1000',newLearnableLayer);
newClassLayer = classificationLayer('Name','new_classoutput');
lgraph = replaceLayer(lgraph,'ClassificationLayer_predictions',newClassLayer);

%
%% Train Network
% The network requires input images of size 224-by-224-by-3, but the  images in the image datastores have different sizes. Use an augmented  image datastore to automatically resize the training images. Specify  additional augmentation operations to perform on the training images, such as  randomly flipping the training images along the vertical axis and randomly  translating them up to 30 pixels horizontally and vertically. Data  augmentation helps prevent the network from overfitting and memorizing  the exact details of the training images.

pixelRange = [-30 30];
imageAugmenter = imageDataAugmenter( ...
'RandXReflection',true, ...
'RandXTranslation',pixelRange, ...
'RandYTranslation',pixelRange);

% To automatically resize the validation images without performing further data augmentation, use an augmented image datastore without specifying  any additional preprocessing operations.

augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain, ...
'DataAugmentation',imageAugmenter);
augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation);

% Specify the training options. For transfer learning, keep the features  from the early layers of the pretrained network (the transferred layer  weights). To slow down learning in the transferred layers, set the  initial learning rate to a small value. Specify the mini-batch size and  validation data. The software validates the network every ValidationFrequency iterations during training.

options = trainingOptions('sgdm', ...
'MiniBatchSize',10, ...
'MaxEpochs',6, ...
'InitialLearnRate',1e-4, ...
'Shuffle','every-epoch', ...
'ValidationData',augimdsValidation, ...
'ValidationFrequency',3, ...
'Verbose',false, ...
'Plots','training-progress');

% Train the network that consists of the transferred and new layers. By default, trainNetwork uses a GPU if one is available (requires Parallel Computing Toolbox™  and a supported GPU device. For more information, see GPU Support by Release).  Otherwise, the network uses a CPU (requires MATLAB Coder Interface for Deep learning Libraries™). You can also specify the execution environment by using the 'ExecutionEnvironment' name-value argument of trainingOptions.

netTransfer = trainNetwork(augimdsTrain,lgraph,options);

%
%% Quantize the Network
% Create a dlquantizer object and specify the network to quantize.

dlquantObj = dlquantizer(netTransfer,'ExecutionEnvironment','FPGA');

%
%% Calibrate the Quantized Network Object
% Use the calibrate function to exercise the network with sample inputs and collect the range information. The calibrate function exercises the network and collects the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. The calibrate function returns a table. Each row of the table contains range information for a learnable parameter of the quantized network.

dlquantObj.calibrate(augimdsTrain)

%
%% Create Target Object
% Use the dlhdl.Target class to create a target object with a custom name for your target device and an interface to connect your target device to the host computer. Interface options are JTAG and Ethernet. To use JTAG,Install Xilinx™ Vivado™ Design Suite 2019.2. To set the Xilinx Vivado toolpath, enter:

% hdlsetuptoolpath('ToolName', 'Xilinx Vivado', 'ToolPath', 'C:\Xilinx\Vivado\2019.2\bin\vivado.bat');

%

hTarget = dlhdl.Target('Xilinx','Interface','Ethernet');

%
%% Create WorkFlow Object
% Use the dlhdl.Workflow class to create an object. When you create the object, specify the network and the bitstream name. Specify the saved pretrained alexnet neural network as the network. Make sure that the bitstream name matches the data type and the FPGA board that you are targeting. In this example, the target FPGA board is the Xilinx ZCU102 SoC board. The bitstream uses a single data type.

hW = dlhdl.Workflow('Network', dlquantObj, 'Bitstream', 'zcu102_int8','Target',hTarget);

%
%% Compile the netTransfer DAG network
%  To compile the netTransfer DAG network, run the compile method of the dlhdl.Workflow object. You can optionally specify the maximum number of input frames.

dn = hW.compile('InputFrameNumberLimit',15)

%
%% Program Bitstream onto FPGA and Download Network Weights
% To deploy the network on the Xilinx ZCU102 hardware, run the deploy function of the dlhdl.Workflow object. This function uses the output of the compile function to program the FPGA board by using the programming file. It also downloads the network weights and biases. The deploy function starts programming the FPGA device, displays progress messages, and the time it takes to deploy the network.

hW.deploy

%
%% Load Image for Prediction
% Load the example image.

imgFile = fullfile(pwd,'MerchData','MathWorks Cube','Mathworks cube_0.jpg');
inputImg = imresize(imread(imgFile),[224 224]);
imshow(inputImg)

%
%% Run Prediction for One Image
% Execute the predict method on the dlhdl.Workflow object and then show the label in the MATLAB command window.

[prediction, speed] = hW.predict(single(inputImg),'Profile','on');
[val, idx] = max(prediction);
dlquantObj.NetworkObject.Layers(end).ClassNames{idx}

% Copyright 2020 The MathWorks, Inc.