Deep Learning Processor Architecture
The software provides a generic deep learning processor IP core that is target-independent and can be deployed to any custom platform that you specify. The processor can be reused and shared to accommodate deep neural networks that have various layer sizes and parameters. Use this processor to rapidly prototype deep neural networks from MATLAB®, and then deploy the network to FPGAs.
This figure shows the deep learning processor architecture.
To illustrate the deep learning processor architecture, consider an image classification example.
DDR External Memory
You can store the input images, the weights, and the output images in the external DDR
memory. The processor consists of four AXI4 Master interfaces that communicate with the
external memory. Using one of the AXI4 Master interfaces, you can load the input images onto
the Block RAM (BRAM). The Block RAM provides the activations to the
Generic Convolution Processor.
Generic Convolution Processor
The Generic Convolution Processor performs the equivalent operation
of one convolution layer. Using another AXI4 Master interface, the weights for the
convolution operation are provided to the Generic Convolution Processor.
The Generic Convolution Processor then performs the convolution operation
on the input image and provides the activations for the Activation
Normalization. The processor is generic because it can support tensors and
shapes of various sizes.
Activation Normalization
Based on the neural network that you provide, the Activation
Normalization module serves the purpose of adding the ReLU nonlinearity, a
maxpool layer, or performs Local Response Normalization (LRN). You see that the processor
has two Activation Normalization units. One unit follows the
Generic Convolution Processor. The other unit follows the
Generic FC Processor.
Conv Controller (Scheduling)
Depending on the number of convolution layers that you have in your pretrained network,
the Conv Controller (Scheduling) acts as ping-pong buffers. The
Generic Convolution Processor and Activation
Normalization can process one layer at a time. To process the next layer, the
Conv Controller (Scheduling) moves back to the BRAM and then performs
the convolution and activation normalization operations for all convolution layers in the
network.
Generic FC Processor
The Generic FC Processor performs the equivalent operation of one
fully-connected layer (FC). Using another AXI4 Master interface, the weights for the
fully-connected layer are provided to the Generic FC Processor. The
Generic FC Processor then performs the fully-connected layer operation
on the input image and provides the activations for the Activation
Normalization module. This processor is also generic because it can support
tensors and shapes of various sizes.
FC Controller (Scheduling)
The FC Controller (Scheduling) works similar to the Conv
Controller (Scheduling). The FC Controller (Scheduling)
coordinates with the FIFO to act as ping-pong buffers for performing the
fully-connected layer operation and Activation Normalization depending on
the number of FC layers, and ReLU, maxpool, or LRN features that you have in your neural
network. After the Generic FC Processor and Activation
Normalization modules process all the frames in
the image, the predictions or scores are transmitted through the AXI4 Master interface and
stored in the external DDR memory.
Deep Learning Processor Applications
One application of the custom deep learning processor IP core is the MATLAB controlled deep learning processor. To create this processor, integrate the deep learning processor IP with the HDL Verifier™ MATLAB as AXI Master IP by using the AXI4 slave interface. Through a JTAG or PCI express interface, you can import various pretrained neural networks from MATLAB, execute the operations specified by the network in the deep learning processor IP, and return the classification results to MATLAB.
For more information, see MATLAB Controlled Deep Learning Processor.