({{, (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net] 
represents a net in which net is folded over a sequence of inputs, maintaining a recurrent state.}, {, (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,{"(Subscript[out, i])"->"(Subscript[in, 1])",…}] 
represents a net in which net is folded over its inputs, maintaining a recurrent state by feeding the (Subscript[out, i]) of each step back to the (Subscript[in, i]) of the next step.}, {, (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,feedback,{"(Subscript[const, 1])","(Subscript[const, 2])",…}]
treats the inputs (Subscript[const, i]) to net as being constant instead of mapped over.}, {, (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,feedback,constants,{"(Subscript[out, 1])",…}]
returns the sequence of values for the specific outputs (Subscript[out, i]).}})

OpenersNotesSection(Spacer1)Details

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net] is equivalent to (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,{"Output"->"State"}] and specifies that the output called "Output" of the folded net will be fed back to the input called "State".

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[…] represents a net that takes one or more sequences and outputs one or more sequences of the same length.

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,{"(Subscript[out, 1])"->"(Subscript[in, 1])",…}] accepts the same inputs as net, except for those (Subscript[in, i]) given by one of the feedback connections "(Subscript[out, i])"->"(Subscript[in, i])".

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,feedback,constants] will map over any inputs that are not specified in feedback or constants. All mapped-over inputs should have the same length.

In the typical case in which net has two inputs named "Input" and "State" and one output named "Output", the single argument form (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net] accepts a single input sequence {(Subscript[x, 1]),(Subscript[x, 2]),…,(Subscript[x, n])} named "Input" and an initial state (Subscript[s, 0]) named "State" and produces a sequence of states {(Subscript[s, 1]),(Subscript[s, 2]),…,(Subscript[s, n])} named "Output", computed by applying the net repeatedly to (Subscript[x, i]) and (Subscript[s, i-1]) to compute (Subscript[s, i ]), effectively using net[<|"Input"->(Subscript[x, i]),"State"->(Subscript[s, i-1])|>].

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,{"(Subscript[out, 1])"->"(Subscript[in, 1])",…}] will have state ports named for all the recurrent ports (Subscript[in, i]) whose values are fed back.

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

Within a (RefLink[NetGraph, paclet:ref/NetGraph]), a connection of the form src->(RefLink[NetPort, paclet:ref/NetPort])[layer,"state"] can be used to provide the initial value of a recurrent state of a (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]). Similarly, a connection of the form (RefLink[NetPort, paclet:ref/NetPort])[layer,"state"]->dst can be used to obtain the final value of a recurrent state. If left unconnected, these states can be tracked by (RefLink[NetStateObject, paclet:ref/NetStateObject]).

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[net,…] can be seen as allowing a form of weight sharing between multiple copies of net, one for each sequence element that is being mapped over.

(RefLink[NetFoldOperator, paclet:ref/NetFoldOperator])[…,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:

OpenersPrimaryExamplesSection(Spacer1)Examples  (7)

OpenersExampleSection0(Spacer1)Basic Examples  (1)

Create a "core" net that takes a state and an input and produces an output:

core=NetGraph[{Plus},{{NetPort["Input"],NetPort["State"]}->1}]

Evaluate the core on an input:

core[<|"Input"->3,"State"->2|>]

Create a (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]) that uses the core to operate on sequences of arrays:

fold=NetFoldOperator[core]

Evaluate the net on an input sequence of length-2 vectors:

fold[{{1,2},{3,4},{5,0},{6,1},{0,0}}]

OpenersExampleSection1(Spacer1)Scope  (5)

Construct a net that takes a weighted average of the state and input:

avg=NetGraph[{ThreadingLayer[0.1*#1+0.9*#2&]},{{NetPort["Input"],NetPort["State"]}->1}]

Apply the net to an input and state:

avg[<|"Input"->0,"State"->1|>]

Use (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]) to create a higher-order net that folds the average over an input sequence:

fold=NetFoldOperator[avg]

Fold the accumulator over a sequence of real numbers:

fold[{1,1,1,1,1}]

Use a (RefLink[NetMapOperator, paclet:ref/NetMapOperator]) to map the folder over the color channels of an image, resulting in the rows of the image being accumulated:

map=NetMapOperator[fold,"Input"->NetEncoder["Image"],"Output"->NetDecoder["Image"]]

Apply the operator to an image:

map[]

Create a net that can take a constant input that is used during the folding operation:

net=NetGraph[{ThreadingLayer[LogisticSigmoid[#1*Cos[#2+#3]]&]},{{NetPort["Input"],NetPort["State"],NetPort["Constant"]}->1},"Input"->"Real"]

fold=NetFoldOperator[net,{"Output"->"State"},{"Constant"}]

Apply the net to an input sequence, providing a value for the constant:

fold[<|"Input"->{1,0,-1,-1,1,1,0},"Constant"->0.5|>]

Create a net that has two independent states that are fed back. These states will additionally be used to provide the final output sequences:

net=NetGraph[{
ThreadingLayer[Tanh[#1+#2*Cos[#3]]&],ThreadingLayer[Tanh[#1-#3*Sin[#2]]&]},{{NetPort["Input"],NetPort["A"],NetPort["B"]}->1->NetPort["X"],{NetPort["Input"],NetPort["A"],NetPort["B"]}->2->NetPort["Y"]},"A"->"Real"]

fold=NetFoldOperator[net,{"X"->"A","Y"->"B"}]

Obtain the output sequences when given the input sequences:

output=fold[{1,0,-1,-1,0,1}]

Construct a net that combines elements from two sequences with a recurrent state using a simple nonlinear function:

net=NetGraph[{ThreadingLayer[#1*#3+#2&]},{{NetPort["A"],NetPort["B"],NetPort["State"]}->1},"A"->"Real"]

fold=NetFoldOperator[net]

Obtain the output sequence when given the input sequences:

fold[<|"A"->{1,0,-1,0,1},"B"->{1,1,-1,0,-1}|>]

Obtain the final state:

fold[<|"A"->{1,0,-1,0,1},"B"->{1,1,-1,0,-1}|>,NetPort[All,"States"]]

Create a (RefLink[NetGraph, paclet:ref/NetGraph]) that allows the initial and final states of a (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]) to be set and obtained:

fold=NetFoldOperator[NetGraph[{Plus},{{NetPort["Input"],NetPort["State"]}->1}]]

graph=NetGraph[{fold},{NetPort["InitialState"]->NetPort[1,"State"],NetPort[1,"State"]->NetPort["FinalState"]}]

Apply the net to an input sequence, providing the initial state:

graph[<|"Input"->{{1,2},{2,0},{3,1}},"InitialState"->{10,0}|>]

The final state is the last element of the output sequence:

%[["FinalState"]]==%[["Output",-1]]

If the initial and final states do not come from a larger network, it is also possible to use (RefLink[NetPort, paclet:ref/NetPort])[(RefLink[All, paclet:ref/All]),"States"] to control these without using a (RefLink[NetGraph, paclet:ref/NetGraph]):

fold[<|"Input"->{{1,2},{2,0},{3,1}},NetPort[All,"States"]-><|"State"->{10,0}|>|>,{"Output",NetPort[All,"States"]}]

OpenersExampleSection3(Spacer1)Applications  (1)

Create training data based on strings containing x's and y's and either (RefLink[Less, paclet:ref/Less]), (RefLink[Greater, paclet:ref/Greater]) or (RefLink[Equal, paclet:ref/Equal]) by comparing the number of x's and y's. The training data consists of all possible sentences up to length 8:

alphabet={"x","y"," "};
inputs=Flatten@Table[StringJoin/@Tuples[alphabet,n],{n,1,8}];
cmpXY[str_]:=Switch[Sign[StringCount[str,"x"]-StringCount[str,"y"]],-1,Less,0,Equal,1,Greater];
outputs=cmpXY/@inputs;
trainingData=Thread[inputs->outputs];
RandomSample[trainingData,4]//InputForm

Define a (RefLink[NetGraph, paclet:ref/NetGraph]) representing the computation performed in a single time step by a basic recurrent layer with a state size of 5 and using a (RefLink[Ramp, paclet:ref/Ramp]) activation:

core=NetGraph[{LinearLayer[5],LinearLayer[5,"Biases"->None],TotalLayer[],Ramp},{NetPort["Input"]->1,NetPort["State"]->2,{1,2}->3->4}]

Allow the previous (RefLink[NetGraph, paclet:ref/NetGraph]) to be applied to a sequence by embedding it in a (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]):

foldRNN=NetFoldOperator[core]

Create a net with the (RefLink[NetFoldOperator, paclet:ref/NetFoldOperator]) defined previously to read an input string and predict one of (RefLink[Less, paclet:ref/Less]), (RefLink[Greater, paclet:ref/Greater]) or (RefLink[Equal, paclet:ref/Equal]):

net=NetChain[{UnitVectorLayer[],foldRNN,SequenceLastLayer[],LinearLayer[3],SoftmaxLayer[]},"Input"->NetEncoder[{"Characters",alphabet}],"Output"->NetDecoder[{"Class",{Less,Equal,Greater}}]];

Train the network:

result=NetTrain[net,trainingData,All,BatchSize->64,MaxTrainingRounds->100,ValidationSet->Scaled[0.2]]

trained=result["TrainedNet"]

Apply the trained network to a list of inputs:

trained[{"xxy","xyy","xyxy"}]

Measure the accuracy on the entire training set:

NetMeasurements[trained,trainingData,"Accuracy"]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[Fold, paclet:ref/Fold])  ▪  (RefLink[NetNestOperator, paclet:ref/NetNestOperator])  ▪  (RefLink[NetMapOperator, paclet:ref/NetMapOperator])  ▪  (RefLink[NetMapThreadOperator, paclet:ref/NetMapThreadOperator])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})