({{, (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers]) 
is an option for (RefLink[NetTrain, paclet:ref/NetTrain]) that specifies learning rate multipliers to apply to specific layers within a (RefLink[NetChain, paclet:ref/NetChain]), (RefLink[NetGraph, paclet:ref/NetGraph]), etc.}})

OpenersNotesSection(Spacer1)Details

With the default value of (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers])->(RefLink[Automatic, paclet:ref/Automatic]), all layers learn at the same rate.

(RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers])->{(Subscript[rule, 1]),(Subscript[rule, 2]),…} specifies a set of rules that will be used to determine learning rate multipliers for every trainable array in the net.

In (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers])->{(Subscript[rule, 1]),(Subscript[rule, 2]),…}, each of the (Subscript[rule, i]) can be of the following forms:

If r is a positive number, it specifies a multiplier to apply to the global learning rate chosen by the training method to determine the learning rate for the given layer or array.

If r is zero or (RefLink[None, paclet:ref/None]), it specifies that the layer or array should not undergo training and will be left unchanged by (RefLink[NetTrain, paclet:ref/NetTrain]).

For each trainable array, the rate used is given by the first matching rule, or 1 if no rule matches.

Rules that specify a subnet (e.g. a nested (RefLink[NetChain, paclet:ref/NetChain]) or (RefLink[NetGraph, paclet:ref/NetGraph])) apply to all layers and arrays within that subnet.

(RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers])->{layer->(RefLink[None, paclet:ref/None])} can be used to "freeze" a specific layer.

(RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers])->{layer->1,_->(RefLink[None, paclet:ref/None])} can be used to "freeze" all layers except for a specific layer.

The hierarchical specification used by (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers]) to refer to parts of a net is equivalent to that used by (RefLink[NetExtract, paclet:ref/NetExtract]) and (RefLink[NetReplacePart, paclet:ref/NetReplacePart]).

The learning rate multipliers used for individual neural net weights can be obtained from a (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) via the property "WeightsLearningRateMultipliers".

OpenersPrimaryExamplesSection(Spacer1)Examples  (3)

OpenersExampleSection0(Spacer1)Basic Examples  (1)

Create and initialize a net with three layers, but train only the last layer:

net=NetInitialize@NetChain[{LinearLayer[3],Ramp,LinearLayer[{}]},"Input"->"Real","Output"->"Real"]

trained=NetTrain[net,{1->1.9,2->4.1,3->6.0,4->8.1},LearningRateMultipliers->{3->1,_->None}]

Evaluate the trained net on an input:

trained[1]

The first layer of the initial net started with zero biases:

NetExtract[net,{1,"Biases"}]

The biases of the first layer remain zero in the trained net:

NetExtract[trained,{1,"Biases"}]

The biases of the third layer have been trained:

NetExtract[trained,{3,"Biases"}]

OpenersExampleSection2(Spacer1)Applications  (1)

Train an existing network to solve a new task. Obtain a pre-trained convolutional model that was trained on handwritten digits:

lenet=NetModel["LeNet Trained on MNIST Data"]

Remove the final two layers, and attach two new layers, in order to classify images into 3 classes:

net=NetJoin[
NetDrop[lenet,-2],
NetChain[{LinearLayer[],SoftmaxLayer[]}]];
net=NetReplacePart[net,"Output"->NetDecoder[{"Class",{"x","y","z"}}]]

Generate training data by rasterizing the characters "x", "y", and "z" with a variety of fonts, sizes, and cases:

letterImage[str_,size_,style_,font_]:=Rasterize[Style[str,style,FontSize->size,FontFamily->font],"Image",ImageSize->{28,28}];

trainingData=Table[
letterImage[case[class],size,style,font]->class,
{class,{"x","y","z"}},
{font,{"Courier","Helvetica","Times New Roman"}},
{style,{Plain,Italic,Bold}},
{size,{7,8,9,10}},
{case,{ToLowerCase,ToUpperCase}}
]//Flatten;

Length[trainingData]

RandomSample[trainingData,10]

Train the modified network on the new task:

trained=NetTrain[net,trainingData,TimeGoal->10,LearningRateMultipliers->{-2->1,_->None},ValidationSet->Scaled[0.1]]

Classify an unseen letter:

x=letterImage["x",11,Italic,"Arial Narrow"]

trained[x,"Probabilities"]

Measure the performance on the original training data, which includes the training and validation set:

NetMeasurements[trained,trainingData,"Accuracy"]

OpenersExampleSection3(Spacer1)Properties & Relations  (1)

Train LeNet on the MNIST dataset with specific learning rate multipliers, returning a (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]):

results=NetTrain[NetModel["LeNet"],ResourceData["MNIST"],All,LearningRateMultipliers->{4;;->2}]

Obtain the actual learning rate multipliers used on individual weight arrays:

results["WeightsLearningRateMultipliers"]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetReplacePart, paclet:ref/NetReplacePart])  ▪  (RefLink[NetExtract, paclet:ref/NetExtract])  ▪  (RefLink[NetPort, paclet:ref/NetPort])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})