({{, (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer])[] 
represents a trainable net layer that normalizes its input data by learning the data mean and variance.}})

OpenersNotesSection(Spacer1)Details and Options

The following optional parameters can be included:

The following learnable arrays can be included:

With (RefLink[Automatic, paclet:ref/Automatic]) settings, the biases, scaling, moving mean, and moving variance arrays are initialized automatically when (RefLink[NetInitialize, paclet:ref/NetInitialize]) or (RefLink[NetTrain, paclet:ref/NetTrain]) is used.

If biases, scaling, moving variance, and moving mean have been added, (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer])[…][input] explicitly computes the output from applying the layer.

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer])[…][{(Subscript[input, 1]),(Subscript[input, 2]),…}] explicitly computes outputs for each of the (Subscript[input, i]).

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

(RefLink[NetExtract, paclet:ref/NetExtract]) can be used to extract biases, scaling, moving variance, and moving mean arrays from a (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) object.

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) is typically used inside (RefLink[NetChain, paclet:ref/NetChain]), (RefLink[NetGraph, paclet:ref/NetGraph]), etc. to regularize and speed up network training.

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) exposes the following ports for use in (RefLink[NetGraph, paclet:ref/NetGraph]) etc.:

When it cannot be inferred from other layers in a larger net, the option "Input"->{(Subscript[n, 1]),(Subscript[n, 2]),…} can be used to fix the input dimensions of (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]).

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) updates the values of "MovingVariance" and "MovingMean" during training with (RefLink[NetTrain, paclet:ref/NetTrain]).

OpenersPrimaryExamplesSection(Spacer1)Examples  (12)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create a (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]):

BatchNormalizationLayer[]

Create an initialized (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) that takes a vector and returns a vector:

batch=NetInitialize@BatchNormalizationLayer["Input"->3]

Apply the layer to an input vector:

batch[{1,2,3}]

OpenersExampleSection1(Spacer1)Scope  (2)

Create an initialized (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) that takes a rank-3 array and returns a rank-3 array:

batch=NetInitialize@BatchNormalizationLayer["Input"->{2,3,3}]

batch[RandomReal[1,{2,3,3}]]//Normal//MatrixForm

Create an initialized (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) that takes a vector and returns a vector:

batch=NetInitialize@BatchNormalizationLayer["Input"->3]

Apply the layer to a batch of input vectors:

batch[{{1,2,3},{4,0.2,3}}]

Use (RefLink[NetEvaluationMode, paclet:ref/NetEvaluationMode]) to use the training behavior of (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]):

batch[{{1,2,3},{4,0.2,3}},NetEvaluationMode->"Train"]

OpenersExampleSection2(Spacer1)Options  (3)

OpenersExampleSubsection0(Spacer1)"Biases"  (1)

Create a (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) with an initial value for the "Biases" parameter:

batch=BatchNormalizationLayer["Biases"->{-1,3.4}]

Extract the "Biases" parameter:

NetExtract[batch,"Biases"]

The default value for "Biases" chosen by (RefLink[NetInitialize, paclet:ref/NetInitialize]) is a zero vector:

batch=NetInitialize@BatchNormalizationLayer["Input"->2];
NetExtract[batch,"Biases"]

OpenersExampleSubsection1(Spacer1)"Epsilon"  (1)

Create a (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) with the "Epsilon" parameter explicitly specified:

batch=BatchNormalizationLayer["Epsilon"->0.1]

Extract the "Epsilon" parameter:

NetExtract[batch,"Epsilon"]

OpenersExampleSubsection2(Spacer1)"Scaling"  (1)

Create an initialized (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) with the "Scaling" parameter set to zero and the "Biases" parameter set to a custom value:

batch=NetInitialize@BatchNormalizationLayer["Scaling"->{0,0,0},"Biases"->{1.3,-22.1,1.2}]

Applying the layer to any input returns the value for the "Biases" parameter:

batch[{1,2,3}]

batch[{-3.4,2.3,100}]

The default value for "Scaling" chosen by (RefLink[NetInitialize, paclet:ref/NetInitialize]) is a vector of 1s:

batch=NetInitialize@BatchNormalizationLayer["Input"->2];
NetExtract[batch,"Scaling"]

OpenersExampleSection3(Spacer1)Applications  (1)

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) is commonly inserted between a (RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer]) and its activation function in order to stabilize and speed up training:

NetChain[{ConvolutionLayer[3,{3,3}],BatchNormalizationLayer["Input"-> {3,28,28}],ElementwiseLayer[Ramp]}]

OpenersExampleSection4(Spacer1)Properties & Relations  (1)

During ordinary evaluation, (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) computes the following function:

batchNormFunction=Function[Block[{sd=Sqrt[#MovingVariance + #Epsilon]},
(#2 * #Scaling / sd )+(#Biases - (#Scaling * #MovingMean)/sd)]];

Evaluate a (RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) on an example vector containing a single channel:

params=<|"Scaling"->{3},"Biases"->{2},"MovingMean"->{1},"MovingVariance"->{2},"Epsilon"->0.001|>;
layer=BatchNormalizationLayer@@Normal[params]

layer[{5}]//Normal

Manually compute the same result:

batchNormFunction[params,{5}]

OpenersExampleSection5(Spacer1)Possible Issues  (3)

Specifying negative values for the "MovingVariance" parameter causes numerical errors during evaluation:

batch=NetInitialize[BatchNormalizationLayer["Input"->{1,2,2},"MovingVariance"->{-2}]]

batch[RandomReal[1,{1,2,2}]]

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) cannot be initialized until all its input and output dimensions are known:

NetInitialize@BatchNormalizationLayer[]

NetInitialize@BatchNormalizationLayer["Input"->3]

(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer]) cannot accept symbolic inputs:

batch=NetInitialize@BatchNormalizationLayer["Input"->2];
batch[{1,2}]

batch[{1,z}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[DropoutLayer, paclet:ref/DropoutLayer])  ▪  (RefLink[NetEvaluationMode, paclet:ref/NetEvaluationMode])  ▪  (RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer])  ▪  (RefLink[PoolingLayer, paclet:ref/PoolingLayer])  ▪  (RefLink[LocalResponseNormalizationLayer, paclet:ref/LocalResponseNormalizationLayer])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetInitialize, paclet:ref/NetInitialize])  ▪  (RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[NetExtract, paclet:ref/NetExtract])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})