ONNXParameters

Parameters of an imported ONNX network for deep learning

Description

ONNXParameters contains the parameters (such as weights and bias) of an imported ONNX™ (Open Neural Network Exchange) network. Use ONNXParameters to perform tasks such as transfer learning.

Creation

Create an ONNXParameters object by using importONNXFunction.

Properties

expand all

`Learnables` — Parameters updated during network training
structure

Parameters updated during network training, specified as a structure. For example, the weights of convolution and fully connected layers are parameters that the network learns during training. To prevent Learnables parameters from being updated during training, convert them to Nonlearnables by using freezeParameters. Convert frozen parameters back to Learnables by using unfreezeParameters.

Add a new parameter to params.Learnables by using addParameter. Remove a parameter from params.Learnables by using removeParameter.

Access the fields of the structure Learnables by using dot notation. For example, params.Learnables.conv1_W could display the weights of the first convolution layer. Initialize the weights for transfer learning by entering params.Learnables.conv1_W = rand([1000,4096]). For more details about assigning a new value and parameter naming, see Tips.

`Nonlearnables` — Parameters unchanged during network training
structure

Parameters unchanged during network training, specified as a structure. For example, padding and stride are parameters that stay constant during training.

Add a new parameter to params.Nonlearnables by using addParameter. Remove a parameter from params.Nonlearnables by using removeParameter.

Access the fields of the structure Nonlearnables by using dot notation. For example, params.Nonlearnables.conv1_Padding could display the padding of the first convolution layer. For more details about parameter naming, see Tips.

`State` — Network state
structure

Network state, specified as a structure. The network State contains information remembered by the network between iterations and updated across multiple training batches. For example, the states of LSTM and batch normalization layers are State parameters.

Add a new parameter to params.State by using addParameter. Remove a parameter from params.State by using removeParameter.

Access the fields of the structure State by using dot notation. For example, params.State.bn1_var could display the variance of the first batch normalization layer. For more details about parameter naming, see Tips.

`NumDimensions` — Number of dimensions for every parameter
structure

This property is read-only.

Number of dimensions for every parameter, specified as a structure. NumDimensions includes trailing singleton dimensions.

Access the fields of the structure NumDimensions by using dot notation. For example, params.NumDimensions.conv1_W could display the number of dimensions for the weights parameter of the first convolution layer.

`NetworkFunctionName` — Name of model function
character vector | string scalar

This property is read-only.

Name of the model function, specified as a character vector or string scalar. The property NetworkFunctionName contains the name of the function NetworkFunctionName, which you specify in importONNXFunction. The function NetworkFunctionName contains the architecture of the imported ONNX network.

Example: 'shufflenetFcn'

Object Functions

`addParameter`	Add parameter to `ONNXParameters` object
`freezeParameters`	Convert learnable network parameters in `ONNXParameters` to nonlearnable
`removeParameter`	Remove parameter from `ONNXParameters` object
`unfreezeParameters`	Convert nonlearnable network parameters in `ONNXParameters` to learnable

Examples

collapse all

Train Imported ONNX Function Using Custom Training Loop

This example uses:

Open Live Script

Import the alexnet convolution neural network as a function and fine-tune the pretrained network with transfer learning to perform classification on a new collection of images.

This example uses several helper functions. To view the code for these functions, see Helper Functions.

Unzip and load the new images as an image datastore. imageDatastore automatically labels the images based on folder names and stores the data as an ImageDatastore object. An image datastore enables you to store large image data, including data that does not fit in memory, and efficiently read batches of images during training of a convolutional neural network. Specify the mini-batch size.

unzip('MerchData.zip');
miniBatchSize = 8;
imds = imageDatastore('MerchData', ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames',...
    'ReadSize', miniBatchSize);

This data set is small, containing 75 training images. Display some sample images.

numImages = numel(imds.Labels);
idx = randperm(numImages,16);
figure
for i = 1:16
    subplot(4,4,i)
    I = readimage(imds,idx(i));
    imshow(I)
end

Extract the training set and one-hot encode the categorical classification labels.

XTrain = readall(imds);
XTrain = single(cat(4,XTrain{:}));
YTrain_categ = categorical(imds.Labels);
YTrain = onehotencode(YTrain_categ,2)';

Determine the number of classes in the data.

classes = categories(YTrain_categ);
numClasses = numel(classes)

numClasses = 5

AlexNet is a convolutional neural network that is trained on more than a million images from the ImageNet database. As a result, the network has learned rich feature representations for a wide range of images. The network can classify images into 1000 object categories, such as keyboard, mouse, pencil, and many animals.

Import the pretrained alexnet network as a function.

alexnetONNX()
params = importONNXFunction('alexnet.onnx','alexnetFcn')

A function containing the imported ONNX network has been saved to the file alexnetFcn.m.
To learn how to use this function, type: help alexnetFcn.

params = 
  ONNXParameters with properties:

             Learnables: [1×1 struct]
          Nonlearnables: [1×1 struct]
                  State: [1×1 struct]
          NumDimensions: [1×1 struct]
    NetworkFunctionName: 'alexnetFcn'

params is an ONNXParameters object that contains the network parameters. alexnetFcn is a model function that contains the network architecture. importONNXFunction saves alexnetFcn in the current folder.

Calculate the classification accuracy of the pretrained network on the new training set.

accuracyBeforeTraining = getNetworkAccuracy(XTrain,YTrain,params);
fprintf('%.2f accuracy before transfer learning\n',accuracyBeforeTraining);

0.01 accuracy before transfer learning

The accuracy is very low.

Display the learnable parameters of the network. These parameters, for example the weights (W) and bias (B) of convolution and fully connected layers, are updated by the network during training. Nonlearnable parameters remain constant during training.

params.Learnables

ans = struct with fields:
    data_Mean: [227×227×3 dlarray]
      conv1_W: [11×11×3×96 dlarray]
      conv1_B: [96×1 dlarray]
      conv2_W: [5×5×48×256 dlarray]
      conv2_B: [256×1 dlarray]
      conv3_W: [3×3×256×384 dlarray]
      conv3_B: [384×1 dlarray]
      conv4_W: [3×3×192×384 dlarray]
      conv4_B: [384×1 dlarray]
      conv5_W: [3×3×192×256 dlarray]
      conv5_B: [256×1 dlarray]
        fc6_W: [6×6×256×4096 dlarray]
        fc6_B: [4096×1 dlarray]
        fc7_W: [1×1×4096×4096 dlarray]
        fc7_B: [4096×1 dlarray]
        fc8_W: [1×1×4096×1000 dlarray]
        fc8_B: [1000×1 dlarray]

The last two learnable parameters of the pretrained network are configured for 1000 classes. The parameters fc8_W and fc8_B must be fine-tuned for the new classification problem. Transfer the parameters to classify 5 classes by initializing them.

params.Learnables.fc8_B = rand(5,1);
params.Learnables.fc8_W = rand(1,1,4096,5);

Freeze all the parameters of the network to convert them to nonlearnable parameters. Because you do not need to compute the gradients of the frozen layers, freezing the weights of many initial layers can significantly speed up network training.

params = freezeParameters(params,'all');

Unfreeze the last two parameters of the network to convert them to learnable parameters.

params = unfreezeParameters(params,'fc8_W');
params = unfreezeParameters(params,'fc8_B');

Now the network is ready for training. Initialize the training progress plot.

plots = "training-progress";
if plots == "training-progress"
    figure
    lineLossTrain = animatedline;
    xlabel("Iteration")
    ylabel("Loss")
end

Specify the training options.

velocity = [];
numEpochs = 5;
miniBatchSize = 16;
numObservations = size(YTrain,2);
numIterationsPerEpoch = floor(numObservations./miniBatchSize);
initialLearnRate = 0.01;
momentum = 0.9;
decay = 0.01;

Train the network.

iteration = 0;
start = tic;
executionEnvironment = "cpu"; % Change to "gpu" to train on a GPU.

% Loop over epochs.
for epoch = 1:numEpochs
    
    % Shuffle data.
    idx = randperm(numObservations);
    XTrain = XTrain(:,:,:,idx);
    YTrain = YTrain(:,idx);
    
    % Loop over mini-batches.
    for i = 1:numIterationsPerEpoch
        iteration = iteration + 1;
        
        % Read mini-batch of data.
        idx = (i-1)*miniBatchSize+1:i*miniBatchSize;
        X = XTrain(:,:,:,idx);        
        Y = YTrain(:,idx);
        
        % If training on a GPU, then convert data to gpuArray.
        if (executionEnvironment == "auto" && canUseGPU) || executionEnvironment == "gpu"
            X = gpuArray(X);         
        end
        
        % Evaluate the model gradients and loss using dlfeval and the
        % modelGradients function.
        [gradients,loss,state] = dlfeval(@modelGradients,X,Y,params);
        params.State = state;
        
        % Determine learning rate for time-based decay learning rate schedule.
        learnRate = initialLearnRate/(1 + decay*iteration);
        
        % Update the network parameters using the SGDM optimizer.
        [params.Learnables,velocity] = sgdmupdate(params.Learnables,gradients,velocity);
        
        % Display the training progress.
        if plots == "training-progress"
            D = duration(0,0,toc(start),'Format','hh:mm:ss');
            addpoints(lineLossTrain,iteration,double(gather(extractdata(loss))))
            title("Epoch: " + epoch + ", Elapsed: " + string(D))
            drawnow
        end
    end
end

Calculate the classification accuracy of the network after fine-tuning.

accuracyAfterTraining = getNetworkAccuracy(XTrain,YTrain,params);
fprintf('%.2f accuracy after transfer learning\n',accuracyAfterTraining);

0.99 accuracy after transfer learning

Helper Functions

This section provides the code of the helper functions used in this example.

The getNetworkAccuracy function evaluates the network performance by calculating the classification accuracy.

function accuracy = getNetworkAccuracy(X,Y,onnxParams)

N = size(X,4);
Ypred = alexnetFcn(X,onnxParams,'Training',false);

[~,YIdx] = max(Y,[],1);
[~,YpredIdx] = max(Ypred,[],1);
numIncorrect = sum(abs(YIdx-YpredIdx) > 0);
accuracy = 1 - numIncorrect/N;

end

The modelGradients function calculates the loss and gradients.

function [grad, loss, state] = modelGradients(X,Y,onnxParams)

[y,state] = alexnetFcn(X,onnxParams,'Training',true);
loss = crossentropy(y,Y,'DataFormat','CB');
grad = dlgradient(loss,onnxParams.Learnables);

end

The alexnetONNX function generates an ONNX model of the alexnet network. You need Deep Learning Toolbox Model for AlexNet Network support to access this model.

function alexnetONNX()
    
exportONNXNetwork(alexnet,'alexnet.onnx');

end

Move Parameters Mislabeled by ONNX Functional Importer

This example uses:

Open Live Script

Import a network saved in the ONNX format as a function, and move the mislabeled parameters by using freeze or unfreeze.

Create an ONNX model from the pretrained alexnet network. Then import alexnet.onnx as a function. Import the pretrained ONNX network using importONNXFunction, which returns an ONNXParamaters object that contains the network parameters. The function also creates a new model function in the current folder that contains the network architecture. Specify the name of the model function as alexnetFcn.

net = alexnet;
exportONNXNetwork(net,'alexnet.onnx');
params = importONNXFunction('alexnet.onnx','alexnetFcn');

A function containing the imported ONNX network has been saved to the file alexnetFcn.m.
To learn how to use this function, type: help alexnetFcn.

importONNXFunction labels the parameters of the imported network as Learnables (parameters that are updated during training) or Nonlearnables (parameters that remain unchanged during training). The labeling is not always accurate. A recommended practice is to check if the parameters are assigned to the correct structure params.Learnables or params.Nonlearnables. Display the learnable and nonlearnable parameters of the imported network.

params.Learnables

ans = struct with fields:
    data_Mean: [227×227×3 dlarray]
      conv1_W: [11×11×3×96 dlarray]
      conv1_B: [96×1 dlarray]
      conv2_W: [5×5×48×256 dlarray]
      conv2_B: [256×1 dlarray]
      conv3_W: [3×3×256×384 dlarray]
      conv3_B: [384×1 dlarray]
      conv4_W: [3×3×192×384 dlarray]
      conv4_B: [384×1 dlarray]
      conv5_W: [3×3×192×256 dlarray]
      conv5_B: [256×1 dlarray]
        fc6_W: [6×6×256×4096 dlarray]
        fc6_B: [4096×1 dlarray]
        fc7_W: [1×1×4096×4096 dlarray]
        fc7_B: [4096×1 dlarray]
        fc8_W: [1×1×4096×1000 dlarray]
        fc8_B: [1000×1 dlarray]

params.Nonlearnables

ans = struct with fields:
            conv1_Stride: [1×2 dlarray]
    conv1_DilationFactor: [1×2 dlarray]
           conv1_Padding: [1×1 dlarray]
          pool1_PoolSize: [1×2 dlarray]
            pool1_Stride: [1×2 dlarray]
           pool1_Padding: [1×1 dlarray]
            conv2_Stride: [1×2 dlarray]
    conv2_DilationFactor: [1×2 dlarray]
           conv2_Padding: [2×2 dlarray]
          pool2_PoolSize: [1×2 dlarray]
            pool2_Stride: [1×2 dlarray]
           pool2_Padding: [1×1 dlarray]
            conv3_Stride: [1×2 dlarray]
    conv3_DilationFactor: [1×2 dlarray]
           conv3_Padding: [2×2 dlarray]
            conv4_Stride: [1×2 dlarray]
    conv4_DilationFactor: [1×2 dlarray]
           conv4_Padding: [2×2 dlarray]
            conv5_Stride: [1×2 dlarray]
    conv5_DilationFactor: [1×2 dlarray]
           conv5_Padding: [2×2 dlarray]
          pool5_PoolSize: [1×2 dlarray]
            pool5_Stride: [1×2 dlarray]
           pool5_Padding: [1×1 dlarray]
              fc6_Stride: [1×2 dlarray]
      fc6_DilationFactor: [1×2 dlarray]
             fc6_Padding: [1×1 dlarray]
              fc7_Stride: [1×2 dlarray]
      fc7_DilationFactor: [1×2 dlarray]
             fc7_Padding: [1×1 dlarray]
              fc8_Stride: [1×2 dlarray]
      fc8_DilationFactor: [1×2 dlarray]
             fc8_Padding: [1×1 dlarray]

Note that params.Learnables contains the parameter data_Mean, which should remain unchanged during training. Convert data_Mean to a nonlearnable parameter. The freezeParameters function removes the parameter data_Mean from param.Learnables and adds it to params.Nonlearnables sequentially.

params = freezeParameters(params,'data_Mean');

Display the updated learnable and nonlearnable parameters.

params.Learnables

ans = struct with fields:
    conv1_W: [11×11×3×96 dlarray]
    conv1_B: [96×1 dlarray]
    conv2_W: [5×5×48×256 dlarray]
    conv2_B: [256×1 dlarray]
    conv3_W: [3×3×256×384 dlarray]
    conv3_B: [384×1 dlarray]
    conv4_W: [3×3×192×384 dlarray]
    conv4_B: [384×1 dlarray]
    conv5_W: [3×3×192×256 dlarray]
    conv5_B: [256×1 dlarray]
      fc6_W: [6×6×256×4096 dlarray]
      fc6_B: [4096×1 dlarray]
      fc7_W: [1×1×4096×4096 dlarray]
      fc7_B: [4096×1 dlarray]
      fc8_W: [1×1×4096×1000 dlarray]
      fc8_B: [1000×1 dlarray]

params.Nonlearnables

ans = struct with fields:
            conv1_Stride: [1×2 dlarray]
    conv1_DilationFactor: [1×2 dlarray]
           conv1_Padding: [1×1 dlarray]
          pool1_PoolSize: [1×2 dlarray]
            pool1_Stride: [1×2 dlarray]
           pool1_Padding: [1×1 dlarray]
            conv2_Stride: [1×2 dlarray]
    conv2_DilationFactor: [1×2 dlarray]
           conv2_Padding: [2×2 dlarray]
          pool2_PoolSize: [1×2 dlarray]
            pool2_Stride: [1×2 dlarray]
           pool2_Padding: [1×1 dlarray]
            conv3_Stride: [1×2 dlarray]
    conv3_DilationFactor: [1×2 dlarray]
           conv3_Padding: [2×2 dlarray]
            conv4_Stride: [1×2 dlarray]
    conv4_DilationFactor: [1×2 dlarray]
           conv4_Padding: [2×2 dlarray]
            conv5_Stride: [1×2 dlarray]
    conv5_DilationFactor: [1×2 dlarray]
           conv5_Padding: [2×2 dlarray]
          pool5_PoolSize: [1×2 dlarray]
            pool5_Stride: [1×2 dlarray]
           pool5_Padding: [1×1 dlarray]
              fc6_Stride: [1×2 dlarray]
      fc6_DilationFactor: [1×2 dlarray]
             fc6_Padding: [1×1 dlarray]
              fc7_Stride: [1×2 dlarray]
      fc7_DilationFactor: [1×2 dlarray]
             fc7_Padding: [1×1 dlarray]
              fc8_Stride: [1×2 dlarray]
      fc8_DilationFactor: [1×2 dlarray]
             fc8_Padding: [1×1 dlarray]
               data_Mean: [227×227×3 dlarray]

Tips

The following rules apply when you assign a new value to a params.Learnables parameter:
- The software automatically converts the new value to a dlarray.
- The new value must be compatible with the existing value of params.NumDimensions.
importONNXFunction derives the field names of the structures Learnables, Nonlearnables, and State from the names in the imported ONNX model file. The field names might differ between imported networks.

Documentation

ONNXParameters

Description

Creation

Properties

`Learnables` — Parameters updated during network training
structure

`Nonlearnables` — Parameters unchanged during network training
structure

`State` — Network state
structure

`NumDimensions` — Number of dimensions for every parameter
structure

`NetworkFunctionName` — Name of model function
character vector | string scalar

Object Functions

Examples

Train Imported ONNX Function Using Custom Training Loop

Move Parameters Mislabeled by ONNX Functional Importer

Tips

See Also

Topics

Deep Learning Toolbox Documentation

Support

Documentation

ONNXParameters

Description

Creation

Properties

Learnables — Parameters updated during network training structure

Nonlearnables — Parameters unchanged during network training structure

State — Network state structure

NumDimensions — Number of dimensions for every parameter structure

NetworkFunctionName — Name of model function character vector | string scalar

Object Functions

Examples

Train Imported ONNX Function Using Custom Training Loop

Move Parameters Mislabeled by ONNX Functional Importer

Tips

See Also

Topics

Deep Learning Toolbox Documentation

Support

`Learnables` — Parameters updated during network training
structure

`Nonlearnables` — Parameters unchanged during network training
structure

`State` — Network state
structure

`NumDimensions` — Number of dimensions for every parameter
structure

`NetworkFunctionName` — Name of model function
character vector | string scalar