({{, (RefLink[NetInitialize, paclet:ref/NetInitialize])[net] 
gives a net in which all uninitialized learnable parameters in net have been given initial values.}, {, (RefLink[NetInitialize, paclet:ref/NetInitialize])[net,(RefLink[All, paclet:ref/All])]
gives a net in which all learnable parameters have been given initial values. }})

OpenersNotesSection(Spacer1)Details and Options

(RefLink[NetInitialize, paclet:ref/NetInitialize])[net,(RefLink[All, paclet:ref/All])] overwrites any existing training or preset learnable parameters in net.

(RefLink[NetInitialize, paclet:ref/NetInitialize]) typically assigns random values to parameters representing weights and zero to parameters representing biases.

The following optional parameters can be included:

Possible settings for (RefLink[Method, paclet:ref/Method]) include:

Suboptions for specific methods can be specified using (RefLink[Method, paclet:ref/Method])->{"method",(Subscript[opt, 1])->(Subscript[val, 1]),…}.

For the method "Xavier", the following suboptions are supported:

For the method "Random", the following suboptions are supported:

For the method "Identity", the following suboption is supported:

For any suboption that expects a distribution, a numeric value stddev can be specified and is taken to mean (RefLink[NormalDistribution, paclet:ref/NormalDistribution])[0,stddev].

By default, all methods initialize bias vectors to zero.

Possible settings for (RefLink[RandomSeeding, paclet:ref/RandomSeeding]) include:

OpenersPrimaryExamplesSection(Spacer1)Examples  (8)

OpenersExampleSection0(Spacer1)Basic Examples  (1)

Create an uninitialized layer:

dot=LinearLayer[2,"Input"->5]

Initialize the layer with random weights:

dot=NetInitialize[dot]

Extract the new initialized weights:

NetExtract[dot,"Weights"]

OpenersExampleSection1(Spacer1)Scope  (1)

Specify "Random" initialization, using normal distributions with a standard deviation of 2 for both weights and biases:

net=NetInitialize[LinearLayer[4,"Input"->2],Method->{"Random","Weights"->2,"Biases"->2}]

Extract the initialized weights and biases:

NetExtract[net,"Weights"]//Normal

NetExtract[net,"Biases"]//Normal

OpenersExampleSection2(Spacer1)Options  (1)

OpenersExampleSubsection0(Spacer1)Method  (1)

Define a network:

net=NetChain[{LinearLayer[200],ElementwiseLayer[Ramp],LinearLayer[200]},"Input"-> 50]

Initialize the net using the "Xavier" initialization:

net2=NetInitialize[net,Method->"Xavier"]

Specify that the "Xavier" method will sample from a uniform distribution:

net3=NetInitialize[net,Method->{"Xavier","Distribution"->"Uniform"}];

Plot a histogram of the weights in the first layer:

Histogram[Flatten@Normal@NetExtract[net3,{1,"Weights"}]]

Specify that the "Xavier" method will sample from a normal distribution:

net3=NetInitialize[net,Method->{"Xavier","Distribution"->"Normal"}];

Plot a histogram of the weights in the first layer:

Histogram[Flatten@Normal@NetExtract[net3,{1,"Weights"}]]

OpenersExampleSection3(Spacer1)Properties & Relations  (2)

(RefLink[NetTrain, paclet:ref/NetTrain]) will automatically call (RefLink[NetInitialize, paclet:ref/NetInitialize]) before training begins. The weights and biases of a simple layer are initialized before training:

net=LinearLayer[1,"Input"->1]

NetExtract[net,"Weights"]

NetExtract[net,"Biases"]

Extract the weights and biases after training:

NetPortGradient

net=NetTrain[net,{{1}->{2},{2}->{3}}]

NetExtract[net,"Weights"]

NetExtract[net,"Biases"]

Create a net that maps vectors of length 1 to vectors of length 1:

net=NetChain[{5,100,5,1},"Input"->1];

Initialize the net using the "Identity" method, which results in a net that attempts to preserve the components of arrays as they pass through linear layers:

net1=NetInitialize[net,Method->"Identity"]

Visualize the output of the net as a function of its input:

Plot[net1[x]//Normal,{x,-1,1},PlotRange->{-1,1}]

Initializing the net using other methods produces a random linear function:

net2=NetInitialize[net,Method->"Xavier"];
Plot[net2[x]//Normal,{x,-1,1},PlotRange->{-1,1}]

OpenersExampleSection4(Spacer1)Possible Issues  (2)

Parameters belonging to certain layers have a fixed initialization method, independent of the (RefLink[Method, paclet:ref/Method]) option in (RefLink[NetInitialize, paclet:ref/NetInitialize]):

batch=BatchNormalizationLayer["Input"-> {2,3,3}];

AssociationMap[Normal@NetExtract[NetInitialize[batch,Method->#],"MovingVariance"]&,{"Xavier","Orthogonal","Identity"}]

By default, (RefLink[NetInitialize, paclet:ref/NetInitialize]) uses (RefLink[RandomSeeding, paclet:ref/RandomSeeding])->(RefLink[Inherited, paclet:ref/Inherited]), which will use the same random seed to initialize the net when (RefLink[NetInitialize, paclet:ref/NetInitialize]) is called repeatedly:

Normal@NetExtract[NetInitialize[LinearLayer[1,"Input"->1]],"Weights"]

Normal@NetExtract[NetInitialize[LinearLayer[1,"Input"->1]],"Weights"]

Use (RefLink[RandomSeeding, paclet:ref/RandomSeeding])->(RefLink[Automatic, paclet:ref/Automatic]) to ensure that repeated calls produce different initializations:

Normal@NetExtract[NetInitialize[LinearLayer[1,"Input"->1],RandomSeeding->Automatic],"Weights"]

Normal@NetExtract[NetInitialize[LinearLayer[1,"Input"->1],RandomSeeding->Automatic],"Weights"]

OpenersExampleSection5(Spacer1)Neat Examples  (1)

Explore how the magnitude of the values used for the weights and biases affects a simple nonlinear net that maps single values to vectors of length 8:

Manipulate[Module[{initNet},
initNet=BlockRandom @ NetInitialize[NetChain[{30,Tanh,8,Tanh},"Input"->"Scalar"],Method->{"Random","Weights"->weights,"Biases"->biases}];
ListLinePlot[
Transpose @Normal@ initNet @ Range[-2,2,.01],GridLines->{{200}},
PlotRange->{-1,1},Ticks->{None,Automatic},
ImageSize->Medium]
],
{{weights,1},0,4},
{{biases,0},0,4}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[NetExtract, paclet:ref/NetExtract])  ▪  (RefLink[RandomVariate, paclet:ref/RandomVariate])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})