({{, (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n] 
represents a trainable recurrent layer that takes a sequence of vectors and produces a sequence of vectors each of size n.}, {, (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n,opts] 
includes options for initial weights and other parameters. }})

OpenersNotesSection(Spacer1)Details and Options

(RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n] represents a net that takes an input matrix representing a sequence of vectors and outputs a sequence of the same length.

Each element of the input sequence is a vector of size k, and each element of the output sequence is a vector of size n.

The size k of the input vectors is usually inferred automatically within a (RefLink[NetGraph, paclet:ref/NetGraph]), (RefLink[NetChain, paclet:ref/NetChain]), etc.

The input and output ports of the net represented by (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n] are:

Given an input sequence {(Subscript[x, 1]),(Subscript[x, 2]),…,(Subscript[x, T])}, a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) outputs a sequence of states {(Subscript[s, 1]),(Subscript[s, 2]),…,(Subscript[s, T])}, where (Subscript[s, t])=(RefLink[Tanh, paclet:ref/Tanh])[(Subscript[w, i]).(Subscript[x, t])+(Subscript[w, s]).(Subscript[s, t-1])+b].

(RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n] has a single state port, "State", which is a vector of size n.

Within a (RefLink[NetGraph, paclet:ref/NetGraph]), a connection of the form src->(RefLink[NetPort, paclet:ref/NetPort])[layer,"State"] can be used to provide the initial value of the state for a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]), corresponding to (Subscript[s, 0]) in the recurrence relation. The default initial value is a zero vector.

Within a (RefLink[NetGraph, paclet:ref/NetGraph]), a connection of the form (RefLink[NetPort, paclet:ref/NetPort])[layer,"State"]->dst can be used to obtain the final value of the state for a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]), corresponding to (Subscript[s, T]) in the recurrence relation.

(RefLink[NetStateObject, paclet:ref/NetStateObject]) can be used to create a net that will remember values for the state of (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) that update when the net is applied to inputs.

An initialized (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[…] that operates on vectors of size k contains the following trainable arrays:

In (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n,opts], initial values can be given to the trainable arrays using a rule of the form "array"->value.

(RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n,"Dropout"->spec] indicates that dropout regularization should be applied during training, in which units are probabilistically set to zero.

Specifying "Dropout"->(RefLink[None, paclet:ref/None]) disables dropout during training.

Specifying "Dropout"->p uses an automatically chosen dropout method having dropout probability p.

Specifying "Dropout"->{"(Subscript[method, 1])"->(Subscript[p, 1]),"(Subscript[method, 2])"->(Subscript[p, 2]),…} can be used to combine specific methods of dropout with the corresponding dropout probabilities. Possible methods include:

The dropout methods "VariationalInput" and "VariationalState" are based on the Gal et al. 2016 method, while "StateUpdate" is based on the Semeniuta et al. 2016 method and "VariationalWeights" is based on the Merity et al. 2017 method.

(RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer])[n,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

OpenersPrimaryExamplesSection(Spacer1)Examples  (11)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) that produces a sequence of length-3 vectors:

BasicRecurrentLayer[3]

Create a randomly initialized (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) that takes a sequence of length-2 vectors and produces a sequence of length-3 vectors:

rnn=NetInitialize@BasicRecurrentLayer[3,"Input"->{"Varying",2}]

Apply the layer to an input sequence:

rnn[{{1,2},{0,1},{2,1},{1,2}}]//MatrixForm

OpenersExampleSection1(Spacer1)Scope  (4)

Create a randomly initialized (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) that takes a string and produces a sequence of length-2 vectors:

rnn=
NetInitialize@BasicRecurrentLayer[2,"Input"->NetEncoder[{"Characters",Automatic,"UnitVector"}]]

Apply the layer to an input string:

rnn["hello world"]//MatrixForm

Thread the layer over a batch of inputs:

rnn[{"good","bye"}]//Map[MatrixForm]

Create a randomly initialized net that takes a sequence of length-2 vectors and produces a single length-3 vector:

net=NetInitialize@NetChain[{
BasicRecurrentLayer[3,"Input"->{"Varying",2}],
SequenceLastLayer[]
}]

Apply the layer to an input:

net[{{1,2},{2,1},{3,0}}]

Thread the layer across a batch of inputs:

net[{{{1,2},{2,1},{3,0}},{{3,3},{0,0}}}]

Create a (RefLink[NetGraph, paclet:ref/NetGraph]) that allows the initial state of a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) to be set:

graph=NetInitialize@NetGraph[
{BasicRecurrentLayer[2,"Input"->{"Varying",2}]},
{NetPort["InitialState"]->NetPort[1,"State"]}]

graph[<|"Input"->{{1,2},{2,0},{3,1}},"InitialState"->{1,1}|>]

Create a (RefLink[NetGraph, paclet:ref/NetGraph]) that allows the final state of a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) to be obtained:

graph=NetInitialize@NetGraph[
{BasicRecurrentLayer[2,"Input"->{"Varying",2}]},
{NetPort[1,"State"]->NetPort["FinalState"]}]

graph[{{1,2},{2,0},{3,1}}]

The final state is the last element of the output sequence:

%[["FinalState"]]==%[["Output",-1]]

OpenersExampleSection2(Spacer1)Options  (2)

OpenersExampleSubsection0(Spacer1)"Dropout"  (2)

Create a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) with the dropout method specified:

BasicRecurrentLayer[2,"Dropout"-> {"VariationalInput"->0.1,"VariationalState"->0.1}]

Create a randomly initialized (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) with specified dropout probability:

rnn=NetInitialize@BasicRecurrentLayer[2,"Dropout"->0.5,"Input"->{"Varying",2}]

Evaluate the layer on a sequence of vectors:

data={{1,2},{3,4},{1,0},{2,2}};
rnn[data]//MatrixForm

Dropout has no effect during evaluation:

rnn2=NetReplacePart[rnn,"Dropout"->0];
rnn[data]==rnn2[data]

Use (RefLink[NetEvaluationMode, paclet:ref/NetEvaluationMode]) to force the training behavior of dropout:

rnn[data,NetEvaluationMode->"Train"]//MatrixForm

Multiple evaluations on the same input can give different results:

Table[rnn[{{1,2},{3,4},{5,6},{7,8}},NetEvaluationMode->"Train"],5]//Map[MatrixPlot[#,ImageSize->30,FrameTicks->None]&]

OpenersExampleSection3(Spacer1)Applications  (2)

Create training data consisting of strings that describe two-digit additions and the corresponding numeric result:

enc=NetEncoder[{"Characters",{DigitCharacter,"+"}}];
makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]->a+b;
trainingData=Flatten@Table[makeRule[i,j],{i,0,99},{j,0,99}];
RandomSample[trainingData,4]//InputForm

Create a network using stacked (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) layers that reads the input string and predicts the numeric result:

net=NetChain[{UnitVectorLayer[],BasicRecurrentLayer[40],BasicRecurrentLayer[40],SequenceLastLayer[],LinearLayer[1]},"Input"->enc,"Output"->"Scalar"];

Train the network:

trained=NetTrain[net,trainingData,BatchSize->64,MaxTrainingRounds->200,ValidationSet->Scaled[0.1]]

Apply the trained network to a list of inputs:

Round@trained[{"5+2","10+25","9+11","44+44"}]

Create training data based on strings containing x's and y's, and either (RefLink[Less, paclet:ref/Less]), (RefLink[Greater, paclet:ref/Greater]) or (RefLink[Equal, paclet:ref/Equal]) by comparing the number of x's and y's. The training data consists of all possible sentences up to length 8:

alphabet={"x","y"," "};
inputs=Flatten@Table[StringJoin/@Tuples[alphabet,n],{n,1,8}];
cmpXY[str_]:=Switch[Sign[StringCount[str,"x"]-StringCount[str,"y"]],-1,Less,0,Equal,1,Greater];
outputs=cmpXY/@inputs;
trainingData=Thread[inputs->outputs];
RandomSample[trainingData,4]//InputForm

Create a network containing a (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]) to read an input string and predict one of (RefLink[Less, paclet:ref/Less]), (RefLink[Greater, paclet:ref/Greater]) or (RefLink[Equal, paclet:ref/Equal]):

net=NetChain[{UnitVectorLayer[],BasicRecurrentLayer[5],SequenceLastLayer[],LinearLayer[3],SoftmaxLayer[]},"Input"->NetEncoder[{"Characters",alphabet}],"Output"->NetDecoder[{"Class",{Less,Equal,Greater}}]];

Train the network:

trained=NetTrain[net,trainingData,BatchSize->64,MaxTrainingRounds->100,ValidationSet->Scaled[0.2]]

Apply the trained network to a list of inputs:

trained[{"xxy","xyy","xyxy"}]

Measure the accuracy on the entire training set:

NetMeasurements[trained,trainingData,"Accuracy"]

OpenersExampleSection4(Spacer1)Properties & Relations  (1)

(RefLink[NetStateObject, paclet:ref/NetStateObject]) can be used to create a net that remembers the state of (RefLink[BasicRecurrentLayer, paclet:ref/BasicRecurrentLayer]):

sobj=NetStateObject[NetInitialize@BasicRecurrentLayer[2,"Input"->{"Varying",2}]]

Each evaluation modifies the state stored inside the (RefLink[NetStateObject, paclet:ref/NetStateObject]):

sobj[{{0,0},{1,1}}]

sobj[{{0,0},{1,1}}]

sobj[{{0,0},{1,1}}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[NetStateObject, paclet:ref/NetStateObject])  ▪  (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer])  ▪  (RefLink[LongShortTermMemoryLayer, paclet:ref/LongShortTermMemoryLayer])  ▪  (RefLink[NetBidirectionalOperator, paclet:ref/NetBidirectionalOperator])  ▪  (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])  ▪  (RefLink[NetMapOperator, paclet:ref/NetMapOperator])  ▪  (RefLink[SequenceLastLayer, paclet:ref/SequenceLastLayer])  ▪  (RefLink[LinearLayer, paclet:ref/LinearLayer])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetExtract, paclet:ref/NetExtract])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})