batchnorm
Normalize each channel of mini-batch
Syntax
Description
The batch normalization operation normalizes each input channel
across a mini-batch. To speed up training of convolutional neural networks and reduce the
sensitivity to network initialization, use batch normalization between convolution and nonlinear
operations such as relu.
Note
This function applies the batch normalization operation to dlarray data. If
you want to apply batch normalization within a layerGraph object
or Layer array, use
the following layer:
[
normalizes each channel of the input mini-batch dlY,mu,sigmaSq] = batchnorm(dlX,offset,scaleFactor)dlX using the mean and
variance statistics computed from each channel and applies a scale factor and offset.
The normalized activation is calculated using the following formula:
where xi is the input activation,
μc (mu) and
σc2
(sigmaSq) are the per-channel mean and variance, respectively, and
ε is a small constant. mu and
sigmaSq are calculated over all 'S' (spatial),
'B' (batch), 'T' (time), and 'U'
(unspecified) dimensions in dlX for each channel.
The normalized activation is offset and scaled according to the following formula:
The offset β and scale factor γ are specified with
the offset and scaleFactor arguments.
The input dlX is a formatted dlarray with
dimension labels. The output dlY is a formatted
dlarray with the same dimension labels as dlX.
[
normalizes each channel of the input mini-batch dlY,datasetMu,datasetSigmaSq] = batchnorm(dlX,offset,scaleFactor,datasetMu,datasetSigmaSq)dlX using the mean and
variance statistics computed from each channel and applies a scale factor and offset. The
function also updates the data set statistics datasetMu and
datasetSigmaSq using the following formula:
where sn is the statistic computed over several mini-batches, sx is the per-channel statistic of the current mini-batch, and ϕ is the decay value for the statistic.
Use this syntax to iteratively update the mean and variance statistics over several mini-batches of data during training. Use the final value of the mean and variance computed over all training mini-batches to normalize data for prediction and classification.
[___] = batchnorm(___,'DataFormat',FMT)
also specifies the dimension format FMT when dlX is
not a formatted dlarray in addition to the input arguments in previous
syntaxes. The output dlY is an unformatted dlarray
with the same dimension order as dlX.
[___] = batchnorm(___,
specifies options using one or more name-value pair arguments in addition to the input
arguments in previous syntaxes. For example, Name,Value)'MeanDecay',3 sets the decay
rate of the moving average computation.
Examples
Normalize Data and Obtain the Statistics
Use batchnorm to normalize each channel of a
mini-batch and obtain the per-channel normalization statistics.
Create the input data as a single observation of random values with a height and width of four and three channels.
height = 4;
width = 4;
channels = 3;
observations = 1;
X = rand(height,width,channels,observations);
dlX = dlarray(X,'SSCB');Create the learnable parameters.
offset = zeros(channels,1); scaleFactor = ones(channels,1);
Compute the batch normalization and obtain the statistics of each channel of the batch.
[dlY,mu,sigmaSq] = batchnorm(dlX,offset,scaleFactor); mu sigmaSq
mu = 3×1
0.6095
0.6063
0.4619
sigmaSq = 3×1
0.1128
0.0880
0.0805
Update Mean and Variance Over Multiple Batches of Data
Use the batchnorm function to normalize several batches of data and update the statistics of the whole data set after each normalization.
Create three batches of data. The data consists of 10-by-10 random arrays with five channels. Each batch contains 20 observations. The second and third batches are scaled by a multiplicative factor of 1.5 and 2.5, respectively, so the mean of the data set increases with each batch.
height = 10; width = 10; channels = 5; observations = 20; X1 = rand(height,width,channels,observations); dlX1 = dlarray(X1,'SSCB'); X2 = 1.5*rand(height,width,channels,observations); dlX2 = dlarray(X2,'SSCB'); X3 = 2.5*rand(height,width,channels,observations); dlX3 = dlarray(X3,'SSCB');
Create the learnable parameters.
offset = zeros(channels,1); scale = ones(channels,1);
Normalize the first batch of data, dlX1, using batchnorm. Obtain the values of the mean and variance of this batch as outputs.
[dlY1,mu,sigmaSq] = batchnorm(dlX1,offset,scale);
Normalize the second batch of data, dlX2. Use mu and sigmaSq as inputs to obtain the values of the combined mean and variance of the data in batches dlX1 and dlX2.
[dlY2,datasetMu,datasetSigmaSq] = batchnorm(dlX2,offset,scale,mu,sigmaSq);
Normalize the final batch of data, dlX3. Update the data set statistics datasetMu and datasetSigmaSq to obtain the values of the combined mean and variance of all data in batches dlX1, dlX2, and dlX3.
[dlY3,datasetMuFull,datasetSigmaSqFull] = batchnorm(dlX3,offset,scale,datasetMu,datasetSigmaSq);
Observe the change in the mean of each channel as each batch is normalized.
plot([mu';datasetMu';datasetMuFull'])
legend({'Channel 1','Channel 2','Channel 3','Channel 4','Channel 5'},'Location','southeast')
xticks([1 2 3])
xlabel('Number of Batches')
xlim([0.9 3.1])
ylabel('Per-Channel Mean')
title('Data Set Mean')
Input Arguments
Output Arguments
More About
Extended Capabilities
See Also
dlarray | dlconv | dlfeval | dlgradient | fullyconnect | groupnorm | relu