%% Import Keras PReLU Layer
% Import layers from a Keras network that has parametric rectified linear unit (PReLU) layers.
% A PReLU layer performs a threshold operation, where for each channel, any input value less than zero is multiplied by a scalar. The PReLU operation is given by
% f\left(x_i \right)=\left\lbrace \begin{array}{ll}
% x_i  & \textrm{if}\;x_i >0\\
% a_i x_i  & \textrm{if}\;x_i \le 0
% \end{array}\right.
% where x_i is the input of the nonlinear activation f on channel i, and a_i is the scaling parameter controlling the slope of the negative part. The subscript i in a_i indicates that the parameter can be a vector and the nonlinear activation can vary on different channels.
% importKerasNetwork and importKerasLayers can import a network that includes PReLU layers. These functions support both scalar-valued and vector-valued scaling parameters. If a scaling parameter is a vector, then the functions replace the vector with the average of the vector elements. You can modify a PReLU layer to have a vector-valued scaling parameter after import.
%
% Specify the network file to import.

modelfile = 'digitsDAGnetwithPReLU.h5';

% digitsDAGnetwithPReLU includes two PReLU layers. One has a scalar-valued scaling parameter, and the other has a vector-valued scaling parameter.
% Import the network architecture and weights from modelfile.

layers = importKerasLayers(modelfile,'ImportWeights',true);

% The importKerasLayers function displays a warning for the PReLu layer p_re_lu_1. The function replaces the vector-valued scaling parameter of p_re_lu_1 with the average of the vector elements. You can change the parameter back to a vector. First, find the index of the PReLU layer by viewing the Layers property.

layers.Layers

% layers has two PReLU layers. Extract the fourth layer p_re_lu_1, which originally had a vector-valued scaling parameter for a channel dimension.

tempLayer = layers.Layers(4)

% The RawAlpha property contains the vector-valued scaling parameter, and the Alpha property contains a scalar that is an element average of the vector values. Reshape RawAlpha to place the vector values in the third dimension, which corresponds to the channel dimension. Then, replace Alpha with the reshaped RawAlpha values.

tempLayer.Alpha = reshape(tempLayer.RawAlpha,[1,1,numel(tempLayer.RawAlpha)])

% Replace the p_re_lu_1 layer in layers with tempLayer.

layers = replaceLayer(layers,'p_re_lu_1', tempLayer);
layers.Layers(4)

% Now the p_re_lu_1 layer has a vector-valued scaling parameter.
% Copyright 2019 The MathWorks, Inc.