Specify Custom Output Layer Backward Loss Function

If Deep Learning Toolbox™ does not provide the layer you require for your classification or regression problem, then you can define your own custom layer. For a list of built-in layers, see List of Deep Learning Layers.

The example Define Custom Weighted Classification Layer shows how to define and create a custom weighted classification output layer with weighted cross entropy loss and goes through the following steps:

Name the layer – Give the layer a name so it can be used in MATLAB^®.
Declare the layer properties – Specify the properties of the layer.
Create a constructor function (optional) – Specify how to construct the layer and initialize its properties. If you do not specify a constructor function, then the software initializes the properties with '' at creation.
Create a forward loss function – Specify the loss between the predictions and the training targets.
Create a backward loss function (optional) – Specify the derivative of the loss with respect to the predictions. If you do not specify a backward loss function, then the forward loss function must support dlarray objects.

Creating a backward loss function is optional. If the forward loss function only uses functions that support dlarray objects, then software determines the derivatives automatically using automatic differentiation. For a list of functions that support dlarray objects, see List of Functions with dlarray Support. If you want to use functions that do not support dlarray objects, or want to use a specific algorithm for the backward loss function, then you can define a custom backward function using this example as a guide.

Create Custom Layer

The example Define Custom Weighted Classification Layer shows how to create a weighted classification layer.

A weighted classification layer computes the weighted cross entropy loss for classification problems. Weighted cross entropy is an error measure between two continuous random variables. For prediction scores Y and training targets T, the weighted cross entropy loss between Y and T is given by

$L = - \frac{1}{N} \sum_{n = 1}^{N} \sum_{i = 1}^{K} w_{i} T_{n i} \log (Y_{n i}),$

where N is the number of observations, K is the number of classes, and w is a vector of weights for each class.

View the layer created in the example Define Custom Weighted Classification Layer. This layer does not have a backwardLoss function.

classdef weightedClassificationLayer < nnet.layer.ClassificationLayer
               
    properties
        % Vector of weights corresponding to the classes in the training
        % data
        ClassWeights
    end

    methods
        function layer = weightedClassificationLayer(classWeights, name)
            % layer = weightedClassificationLayer(classWeights) creates a
            % weighted cross entropy loss layer. classWeights is a row
            % vector of weights corresponding to the classes in the order
            % that they appear in the training data.
            % 
            % layer = weightedClassificationLayer(classWeights, name)
            % additionally specifies the layer name. 

            % Set class weights
            layer.ClassWeights = classWeights;

            % Set layer name
            if nargin == 2
                layer.Name = name;
            end

            % Set layer description
            layer.Description = 'Weighted cross entropy';
        end
        
        function loss = forwardLoss(layer, Y, T)
            % loss = forwardLoss(layer, Y, T) returns the weighted cross
            % entropy loss between the predictions Y and the training
            % targets T.

            N = size(Y,4);
            Y = squeeze(Y);
            T = squeeze(T);
            W = layer.ClassWeights;
    
            loss = -sum(W*(T.*log(Y)))/N;
        end
    end
end

Create Backward Loss Function

Implement the backwardLoss function that returns the derivatives of the loss with respect to the input data and the learnable parameters.

The syntax for backwardLoss is dLdY = backwardLoss(layer, Y, T). The input Y contains the predictions made by the network and T contains the training targets. The output dLdY is the derivative of the loss with respect to the predictions Y. The output dLdY must be the same size as the layer input Y.

The dimensions of Y and T are the same as the inputs in forwardLoss.

The derivative of the weighted cross entropy loss with respect to the predictions Y is given by

$\frac{δ L}{δ Y_{i}} = - \frac{1}{N} \frac{w_{i} T_{i}}{Y_{i}},$

where N is the number of observations and w is a vector of weights for each class.

Create the backward loss function that returns these derivatives.

        function dLdY = backwardLoss(layer, Y, T)
            % dLdY = backwardLoss(layer, Y, T) returns the derivatives of
            % the weighted cross entropy loss with respect to the
            % predictions Y.
            
            [~,~,K,N] = size(Y);
            Y = squeeze(Y);
            T = squeeze(T);
            W = layer.ClassWeights;
			
            dLdY = -(W'.*T./Y)/N;
            dLdY = reshape(dLdY,[1 1 K N]);
        end

Complete Layer

View the completed layer class file.

classdef weightedClassificationLayer < nnet.layer.ClassificationLayer
               
    properties
        % Vector of weights corresponding to the classes in the training
        % data
        ClassWeights
    end

    methods
        function layer = weightedClassificationLayer(classWeights, name)
            % layer = weightedClassificationLayer(classWeights) creates a
            % weighted cross entropy loss layer. classWeights is a row
            % vector of weights corresponding to the classes in the order
            % that they appear in the training data.
            % 
            % layer = weightedClassificationLayer(classWeights, name)
            % additionally specifies the layer name. 

            % Set class weights
            layer.ClassWeights = classWeights;

            % Set layer name
            if nargin == 2
                layer.Name = name;
            end

            % Set layer description
            layer.Description = 'Weighted cross entropy';
        end
        
        function loss = forwardLoss(layer, Y, T)
            % loss = forwardLoss(layer, Y, T) returns the weighted cross
            % entropy loss between the predictions Y and the training
            % targets T.

            N = size(Y,4);
            Y = squeeze(Y);
            T = squeeze(T);
            W = layer.ClassWeights;
    
            loss = -sum(W*(T.*log(Y)))/N;
        end
        
                function dLdY = backwardLoss(layer, Y, T)
            % dLdY = backwardLoss(layer, Y, T) returns the derivatives of
            % the weighted cross entropy loss with respect to the
            % predictions Y.
            
            [~,~,K,N] = size(Y);
            Y = squeeze(Y);
            T = squeeze(T);
            W = layer.ClassWeights;
			
            dLdY = -(W'.*T./Y)/N;
            dLdY = reshape(dLdY,[1 1 K N]);
        end
    end
end

GPU Compatibility

If the layer forward functions fully support dlarray objects, then the layer is GPU compatible. Otherwise, to be GPU compatible, the layer functions must support inputs and return outputs of type gpuArray (Parallel Computing Toolbox).

Many MATLAB built-in functions support gpuArray (Parallel Computing Toolbox) and dlarray input arguments. For a list of functions that support dlarray objects, see List of Functions with dlarray Support. For a list of functions that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox). To use a GPU for deep learning, you must also have a CUDA^® enabled NVIDIA^® GPU with compute capability 3.0 or higher. For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).

Documentation