({{, (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[] 
represents a trainable net layer that normalizes its input data across the second and subsequent dimensions and applies an independent scaling and bias to each component of the first dimension.}, {, (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[aggregationlevels]
normalizes its input data across the specified aggregation levels and applies a learned scaling and bias on the remaining levels.}, {, (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[aggregationlevels,scalinglevels]
applies a learned scaling and bias at the specified scaling levels.}})

OpenersNotesSection(Spacer1)Details and Options

In (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[aggregationlevels], the aggregation levels determine which levels of the input are aggregated over to compute the mean and variance statistics used to normalize. The aggregation levels can be specified using the following settings:

In (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[], the aggregation levels used are 2;;(RefLink[All, paclet:ref/All]).

In (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[aggregationlevels,scalinglevels], the scaling levels determine which levels of the input have component-wise specific scaling values and biases applied to them. The scaling levels can be specified using the following settings:

The following optional parameters can be included:

The following learnable parameters can be specified:

With (RefLink[Automatic, paclet:ref/Automatic]) settings, scaling and bias parameters are initialized automatically when (RefLink[NetInitialize, paclet:ref/NetInitialize]) or (RefLink[NetTrain, paclet:ref/NetTrain]) is used. With (RefLink[None, paclet:ref/None]) settings, scaling and adding biases can be disabled.

If scaling and bias parameters have been initialized or disabled, (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[…][input] explicitly computes the output from applying the layer.

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[…][input] explicitly computes the output from applying the layer to input.

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[…][{(Subscript[input, 1]),(Subscript[input, 2]),…}] explicitly computes outputs for each of the (Subscript[input, i]).

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

When it cannot be inferred from other layers in a larger net, the option "Input"->{(Subscript[d, 1]),(Subscript[d, 2]),…} can be used to fix the input dimensions of (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]).

(RefLink[NetExtract, paclet:ref/NetExtract]) can be used to extract the scaling, biases and epsilon parameters from a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) object.

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) is typically used inside (RefLink[NetChain, paclet:ref/NetChain]), (RefLink[NetGraph, paclet:ref/NetGraph]), etc.

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) exposes the following ports for use in (RefLink[NetGraph, paclet:ref/NetGraph]) etc.:

Consider an input array x that can be decomposed into (Subscript[d, a])×(Subscript[d, b])×… subarrays (Subscript[x, A]) across the aggregation levels {a,b,…} and into (Subscript[d, i])×(Subscript[d, j])×… subarrays (Subscript[x, I]) across the scaling levels {i,j,…}. The scaling and biases parameters γ and β have dimensions ((Subscript[d, i])×(Subscript[d, j])×…). Then the output, with the same dimensions as the input array, is obtained via (Subscript[o, I])=(Subscript[γ, I])((Subscript[x, I])-(Subscript[μ, I]))/(Sqrt[((Subscript[σ, I])^2)+ϵ])+(Subscript[β, I]), where the mean arrays are given by (Subscript[μ, i])=(∑_( A)^( ))(Subscript[x, A,i])/((Subscript[d, a])× (Subscript[d, b])×…) and the variance by ((Subscript[σ, i])^2)=(∑_( A)^( ))((((Subscript[x, A,i])-(Subscript[μ, i])))^2)/((Subscript[d, a])× (Subscript[d, b])×…).

The default (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[] corresponds to instance normalization in Ulyanov et al., "Instance Normalization: The Missing Ingredient for Fast Stylization", 2016, with the channel dimension at the first level and spatial dimensions at subsequent levels.

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer])[2;;(RefLink[All, paclet:ref/All]),"Same","Input"->{"Varying",Automatic}] corresponds to layer normalization in Ba et al., "Layer Normalization", 2016, with the channel dimension at the last level and the time dimension at the first level.

OpenersPrimaryExamplesSection(Spacer1)Examples  (11)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]):

NormalizationLayer[]

Create an initialized (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) that takes a rank-3 array:

norm=NetInitialize@NormalizationLayer["Input"->{2,2,3}]

Apply the layer to an input:

output=norm[{{{1,2,3},{4,5,6}},{{3,2,1},{1,2,3}}}];

MatrixForm/@output

The output is a list of arrays with values centered around zero:

Mean/@Flatten/@output

The standard deviations of the output arrays are roughly unitary:

StandardDeviation/@Flatten/@output

OpenersExampleSection1(Spacer1)Scope  (3)

Create an instance normalization layer that takes an RGB image and returns an RGB image:

instancenorm=NetInitialize@NormalizationLayer["Input"->NetEncoder["Image"],"Output"->NetDecoder["Image"]]

Apply the layer to an image:

instancenorm[]

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) automatically threads over batches of inputs:

instancenorm[{,}]

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) that takes a sequence of two-dimensional vectors and returns the sequence of normalized vectors scaled by a factor of 2 and shifted by -1:

layernorm=NormalizationLayer[2,"Same","Input"->{"Varying",2},"Scaling"->2,"Biases"->-1]

Apply layer normalization to some sinusoidal input:

input=Table[Cos[x*s]*x,{x,Range[0,1,0.01]},{s,{2*Pi,4*Pi}}];

output=layernorm[input];

Visualize the values before and after normalization:

ListLinePlot[Transpose@input]

ListLinePlot[Transpose@output]

Create an initialized (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) that takes a rank-6 array and normalize across all dimensions:

norm=NetInitialize@NormalizationLayer[All,"Same","Input"->{1,2,3,4,5,6}]

Apply the layer to an input:

Mean@Flatten@norm[RandomReal[1,{1,2,3,4,5,6}]]

StandardDeviation@Flatten@norm[RandomReal[1,{1,2,3,4,5,6}]]

OpenersExampleSection2(Spacer1)Options  (5)

OpenersExampleSubsection0(Spacer1)"Biases"  (2)

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) with the biases explicitly specified:

norm=NormalizationLayer["Biases"->{2.1,3.4}]

Extract the biases:

NetExtract[norm,"Biases"]

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) without any bias:

norm=NormalizationLayer["Biases"->None]

The layer does not have any learnable bias:

NetExtract[norm,"Biases"]

OpenersExampleSubsection1(Spacer1)"Epsilon"  (1)

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) with the "Epsilon" parameter explicitly specified:

norm=NormalizationLayer["Epsilon"->0.1]

Extract the "Epsilon" parameter:

NetExtract[norm,"Epsilon"]

OpenersExampleSubsection2(Spacer1)"Scaling"  (2)

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) with the scaling parameter explicitly specified:

norm=NormalizationLayer["Scaling"->{-1.2,2.3,4.5}]

Extract the scaling parameters:

NetExtract[norm,"Scaling"]

Create a (RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) without any scaling:

norm=NormalizationLayer["Scaling"->None,"Biases"->None]

The layer does not have any learnable scaling parameter:

NetExtract[norm,"Scaling"]

OpenersExampleSection3(Spacer1)Possible Issues  (1)

(RefLink[NormalizationLayer, paclet:ref/NormalizationLayer]) cannot be initialized until all its input and output dimensions are known:

NetInitialize@NormalizationLayer[]

NetInitialize@NormalizationLayer["Input"->{2,55,55}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[BatchNormalizationLayer, paclet:ref/BatchNormalizationLayer])  ▪  (RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer])  ▪  (RefLink[LocalResponseNormalizationLayer, paclet:ref/LocalResponseNormalizationLayer])  ▪  (RefLink[AggregationLayer, paclet:ref/AggregationLayer])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetInitialize, paclet:ref/NetInitialize])  ▪  (RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[NetExtract, paclet:ref/NetExtract])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})