Deep Learning Custom Training Loops

Customize deep learning training loops and loss functions

If the trainingOptions function does not provide the training options that you need for your task, or custom output layers do not support the loss functions that you need, then you can define a custom training loop. For networks that cannot be created using layer graphs, you can define custom networks as a function. To learn more, see Define Custom Training Loops, Loss Functions, and Networks.

Functions

expand all

dlnetworkDeep learning network for custom training loops
forwardCompute deep learning network output for training
predictCompute deep learning network output for inference
adamupdateUpdate parameters using adaptive moment estimation (Adam)
rmspropupdate Update parameters using root mean squared propagation (RMSProp)
sgdmupdate Update parameters using stochastic gradient descent with momentum (SGDM)
dlupdate Update parameters using custom function
minibatchqueueCreate mini-batches for deep learning
onehotencodeEncode data labels into one-hot vectors
onehotdecodeDecode probability vectors into class labels
dlarrayDeep learning array for custom training loops
dlgradientCompute gradients for custom training loops using automatic differentiation
dlfevalEvaluate deep learning model for custom training loops
dimsDimension labels of dlarray
finddimFind dimensions with specified label
stripdimsRemove dlarray labels
extractdataExtract data from dlarray
isdlarrayDetermine whether input is dlarray
functionToLayerGraphConvert deep learning model function to a layer graph
dlconvDeep learning convolution
dltranspconvDeep learning transposed convolution
lstmLong short-term memory
gruGated recurrent unit
embedEmbed discrete data
fullyconnectSum all weighted input data and apply a bias
reluApply rectified linear unit activation
leakyreluApply leaky rectified linear unit activation
batchnormNormalize each channel of mini-batch
crosschannelnormCross channel square-normalize using local responses
groupnormNormalize activations across groups of channels
avgpoolPool data to average values over spatial dimensions
maxpoolPool data to maximum value
maxunpoolUnpool the output of a maximum pooling operation
softmaxApply softmax activation to channel dimension
crossentropyCross-entropy loss for classification tasks
sigmoidApply sigmoid activation
mseHalf mean squared error

Topics

Custom Training Loops

Training Deep Learning Models in MATLAB

Learn how to training deep learning models in MATLAB®.

Define Custom Training Loops, Loss Functions, and Networks

Learn how to define and customize deep learning training loops, loss functions, and networks using automatic differentiation.

Train Network Using Custom Training Loop

This example shows how to train a network that classifies handwritten digits with a custom learning rate schedule.

Specify Training Options in Custom Training Loop

Learn how to specify common training options in a custom training loop.

Define Model Gradients Function for Custom Training Loop

Learn how to define a model gradients function for a custom training loop.

Update Batch Normalization Statistics in Custom Training Loop

This example shows how to update the network state in a custom training loop.

Make Predictions Using dlnetwork Object

This example shows how to make predictions using a dlnetwork object by splitting data into mini-batches.

Train Network on Image and Feature Data

This example shows how to train a network that classifies handwritten digits using both image and feature input data.

Train Network with Multiple Outputs

This example shows how to train a deep learning network with multiple outputs that predict both labels and angles of rotations of handwritten digits.

Model Functions

Train Network Using Model Function

This example shows how to create and train a deep learning network by using functions rather than a layer graph or a dlnetwork.

Update Batch Normalization Statistics Using Model Function

This example shows how to update the network state in a network defined as a function.

Make Predictions Using Model Function

This example shows how to make predictions using a model function by splitting data into mini-batches.

Initialize Learnable Parameters for Model Functions

Learn how to initialize learnable parameters for custom training loops using a model function.

Automatic Differentiation

List of Functions with dlarray Support

View the list of functions that support dlarray objects.

Automatic Differentiation Background

Learn how automatic differentiation works.

Use Automatic Differentiation In Deep Learning Toolbox

How to use automatic differentiation in deep learning.

Featured Examples

Train Generative Adversarial Network (GAN)

Train Generative Adversarial Network (GAN)

Train a generative adversarial network (GAN) to generate images.

Open Live Script
Train Conditional Generative Adversarial Network (CGAN)

Train Conditional Generative Adversarial Network (CGAN)

Train a conditional generative adversarial network (CGAN) to generate images.

Open Live Script
Train Fast Style Transfer Network

Train Fast Style Transfer Network

Train a network to transfer the style of an image to a second image. It is based on the architecture defined in [1].

Open Live Script
Train Network Using Custom Training Loop

Train Network Using Custom Training Loop

Train a network that classifies handwritten digits with a custom learning rate schedule.

Open Live Script
Train Network Using Model Function

Train Network Using Model Function

Create and train a deep learning network by using functions rather than a layer graph or a dlnetwork. The advantage of using functions is the flexibility to describe a wide variety of networks. The disadvantage is that you must complete more steps and prepare your data carefully. This example uses images of handwritten digits, with the dual objectives of classifying the digits and determining the angle of each digit from the vertical.

Open Live Script
Train a Siamese Network for Dimensionality Reduction

Train a Siamese Network for Dimensionality Reduction

Train a Siamese network to compare handwritten digits using dimensionality reduction.

Open Live Script
Train a Siamese Network to Compare Images

Train a Siamese Network to Compare Images

Train a Siamese network to identify similar images of handwritten characters.

Open Live Script
Image Captioning Using Attention

Image Captioning Using Attention

Train a deep learning model for image captioning using attention.

Open Live Script
Multilabel Text Classification Using Deep Learning

Multilabel Text Classification Using Deep Learning

Classify text data that has multiple independent labels.

Open Live Script
Sequence-to-Sequence Translation Using Attention

Sequence-to-Sequence Translation Using Attention

Convert decimal strings to Roman numerals using a recurrent sequence-to-sequence encoder-decoder model with attention.

Open Live Script
Sequence-to-Sequence Classification Using 1-D Convolutions

Sequence-to-Sequence Classification Using 1-D Convolutions

Classify each time step of sequence data using a generic temporal convolutional network (TCN).

Open Live Script
Train Network in Parallel with Custom Training Loop

Train Network in Parallel with Custom Training Loop

Set up a custom training loop to train a network in parallel. In this example, parallel workers train on portions of the overall mini-batch. If you have a GPU, then training happens on the GPU. During training, a DataQueue object sends training progress information back to the MATLAB client.

Open Live Script
Train Variational Autoencoder (VAE) to Generate Images

Train Variational Autoencoder (VAE) to Generate Images

Create a variational autoencoder (VAE) in MATLAB to generate digit images. The VAE generates hand-drawn digits in the style of the MNIST data set.

Open Live Script
Train Network Using Cyclical Learning Rate for Snapshot Ensembling

Train Network Using Cyclical Learning Rate for Snapshot Ensembling

Train a network to classify images of objects using a cyclical learning rate schedule and snapshot ensembling for better test accuracy. In the example, you learn how to use a cosine function for the learning rate schedule, take snapshots of the network during training to create a model ensemble, and add L2-norm regularization (weight decay) to the training loss.

Open Live Script

Deep Learning Toolbox Documentation

Support