Sequence Learning and NLP with Neural Networks

Sequence learning refers to a variety of related tasks that neural nets can be trained to perform. What all these tasks have in common is that the input to the net is a sequence of some kind. This input is usually variable length, meaning that the net can operate equally well on short or long sequences.

An unencoded variable-length array, indicated visually as a placeholder such as n in e.g. ({{}}).

A variable-length array, encoded from a string using a (StringTypeLink["Characters", paclet:ref/netencoder/Characters]) or (StringTypeLink["Tokens", paclet:ref/netencoder/Tokens]) (RefLink[NetEncoder, paclet:ref/NetEncoder]).

A variable-length array, encoded from an (RefLink[Audio, paclet:ref/Audio]) object using an (StringTypeLink["Audio", paclet:ref/netencoder/Audio]), (StringTypeLink["AudioSpectrogram", paclet:ref/netencoder/AudioSpectrogram]), (StringTypeLink["AudioSTFT", paclet:ref/netencoder/AudioSTFT]), etc. (RefLink[NetEncoder, paclet:ref/NetEncoder]).

Fixed-length forms of the above, e.g. by using the "TargetLength" option to the (RefLink[NetEncoder, paclet:ref/NetEncoder]).

What distinguishes the various sequence learning tasks is the form of the output of the net. Here, there is wide diversity of techniques, with corresponding forms of output:

For autoregressive language models, used to model the probability of p(x) of a particular sequence x, the output is the next element of the sequence. In the case of a textual model, this is a character or token, as decoded via a (StringTypeLink["Class", paclet:ref/netencoder/Class]), (StringTypeLink["Characters", paclet:ref/netencoder/Characters]) or "Token" (RefLink[NetDecoder, paclet:ref/NetDecoder]).

For sequence tagging models, the output is a sequence of classes of the same length as the input. For example, in the case of a part-of-speech tagger, these classes are "Noun", "Verb", etc. For this, a (StringTypeLink["Class", paclet:ref/netencoder/Class]) (RefLink[NetDecoder, paclet:ref/NetDecoder]) is appropriate.

For translation models, e.g. an English to French translator, the output is itself a language model, albeit one that is conditional on the source sequence. In other words, there are two inputs to the net: the complete source sequence, and the target sequence so far, and the output is a prediction of the next element of the target sequence.

For CTC models, the input sequence is used to form a sequence of intermittent predictions for the target sequence, which is always shorter than the input sequence. Examples of this include handwriting recognition from pixel or stroke data, in which the input is segmented into individual characters, or audio transcription, in which features of the audio are segmented into characters or phonemes. For these, a (StringTypeLink["CTCBeamSearch", paclet:ref/netdecoder/CTCBeamSearch]) (RefLink[NetDecoder, paclet:ref/NetDecoder]) must be used.

We give simple examples of most of these techniques in this tutorial.

Sequence Regression

Integer Addition

In this example, a net is trained to add two two-digit integers together. What makes the problem hard is that the inputs are strings rather than numeric values, whereas the output is a numeric value. For example, the net takes as input "25+33" and needs to return a real number close to 58. Note that the input is variable length, as the training data contains examples of length 3 ("5+5"), length 4 ("5+10") and length 5 ("10+10").

Create training data based on strings that describe two-digit additions and the corresponding numeric result:

enc=NetEncoder[{"Characters",{DigitCharacter,"+"}}];
makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]->a+b;
trainingData=Flatten@Table[makeRule[i,j],{i,0,99},{j,0,99}];
RandomSample[trainingData,4]//InputForm

Create a net composed of a chain of recurrent layers to read an input string and predict the numeric result:

net=NetChain[{UnitVectorLayer[],LongShortTermMemoryLayer[40],LongShortTermMemoryLayer[20],SequenceLastLayer[],LinearLayer[]},"Input"->enc,"Output"->"Real"];

Train the network:

result=NetTrain[net,trainingData,All,BatchSize->64,MaxTrainingRounds->200,ValidationSet->Scaled[0.1]]
trained=result["TrainedNet"]

Apply the trained network to a list of inputs:

Round@trained[{"5+2","10+25","9+11","44+44"}]

Sequence Classification

Sentiment Analysis

Train a classifier that classifies movie review snippets as "positive" or "negative".

First, obtain the training and test data:

trainingData=ExampleData[{"MachineLearning","MovieReview"},"TrainingData"];
testData=ExampleData[{"MachineLearning","MovieReview"},"TestData"];

RandomSample[trainingData,2]

Define a net that takes a string of words as input and returns either "positive" or "negative":

net=NetChain[{EmbeddingLayer[40],DropoutLayer[0.3],LongShortTermMemoryLayer[20],SequenceLastLayer[],LinearLayer[2],SoftmaxLayer[]},
"Input"->NetEncoder[{"Tokens"}],"Output"-> NetDecoder[{"Class",{"positive","negative"}}]
]

Train the net for two training rounds:

result=NetTrain[net,trainingData,All,ValidationSet->testData,MaxTrainingRounds->2]
trained=result["TrainedNet"]

Evaluate the trained net on a example from the test set, obtaining the probabilities:

ex=testData[[50,1]]

trained[ex,"Probabilities"]

Sequence-to-Sequence Learning

Sequence-to-sequence learning is a learning task where both the input and the predicted output are sequences. Tasks such as translating German to English, transcribing speech from an audio file, sorting lists of numbers, etc. are examples of this task.

Integer Addition with Fixed-Length Output

This example demonstrates how to train nets that take a variable-length sequence as input and predict a fixed-length sequence as output. We take a string that describes a sum, e.g. "250+123", and produce an output string that describes the answer, e.g. "373".

Create training data based on strings that describe three-digit additions and the corresponding result as a string. In order for the output to be fixed length, all outputs are padded to the maximum length of 4 (as the maximum value is 999+999=1998):

makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]->IntegerString[a+b];
data=RandomSample[Flatten@Table[makeRule[i,j],{i,0,999},{j,0,999}],40000];
data[[;;,2]]=StringPadRight[data[[;;,2]],4];
{testData,trainingData}=TakeDrop[data,4000];

RandomSample[trainingData,4]//InputForm

Define a net, taking a string as input and returning another string of length 4:

net=NetChain[{UnitVectorLayer[11],LongShortTermMemoryLayer[128],SequenceLastLayer[],ReplicateLayer[4],LongShortTermMemoryLayer[128],NetMapOperator[LinearLayer[]],SoftmaxLayer[]},
"Input"->NetEncoder[{"Characters",{DigitCharacter,"+"}}],"Output"->NetDecoder[{"Characters",{DigitCharacter," "}}]]

Train the net for 40 training rounds:

result=NetTrain[net,trainingData,All,ValidationSet->testData,MaxTrainingRounds->40]
trainedNet=result["TrainedNet"]

Apply the net to some examples:

StringTrim@trainedNet[{"2+432","100+43","400+500"}]

Obtain the accuracy of the net on the test set:

prediction=trainedNet[testData[[;;,1]]];
accuracy=N@Count[prediction-testData[[;;,2]],0]/Length[prediction]

Integer Addition with Variable-Length Output

This example demonstrates how to train nets on sequences where both the input and output are variable-length sequences, and those sequences are not the same. One sophisticated example of this task is translating English to German, but the example we cover is a simpler problem: taking a string that describes a sum, e.g. "250+123", and producing an output string that describes the answer, e.g. "373". The method used is based on I. Sutskever et al., "Sequence to Sequence Learning with Neural Networks", 2014.

Create training data based on strings that describe three-digit additions and the corresponding result as a string:

makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]->IntegerString[a+b];
data=Flatten@Table[makeRule[i,j],{i,0,999},{j,0,999}];
data=RandomSample@Catenate@GroupBy[data,First/*StringLength,RandomSample[#,UpTo[10000]]&];
{testData,trainingData}=TakeDrop[data,2500];

RandomSample[trainingData,4]

Create a (RefLink[NetEncoder, paclet:ref/NetEncoder]) that uses a special code for the start and end of a string, which will be used to indicate the beginning and end of the output sequence:

targetEnc=NetEncoder[{"Characters",{DigitCharacter,{StartOfString,EndOfString}->Automatic}}]

Evaluate the encoder on an input string:

targetEnc["1"]

Create a similar encoder for the input, which contains a "+":

inputEnc=NetEncoder[{"Characters",{DigitCharacter,"+"}}]

Define a net that takes a sequence of inputs and returns a single vector of size 150 as output:

encoderNet=NetChain[{UnitVectorLayer[],GatedRecurrentLayer[150],SequenceLastLayer[]}]

Define a net that takes an input vector of size 150 and a sequence of vectors as input and returns a sequence of vectors as output:

decoderNet=NetGraph[{
UnitVectorLayer[11],
SequenceMostLayer[],
GatedRecurrentLayer[150],
NetMapOperator[LinearLayer[]],
SoftmaxLayer[]},
{NetPort["Input"]->1->2->3->4->5,
NetPort["State"]->NetPort[3,"State"]}
]

Define a net with a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) and containing the encoder and decoder nets:

trainingNet=NetGraph[<|"encoder"->encoderNet,"decoder"->decoderNet,"loss"->CrossEntropyLossLayer["Index"],"rest"->SequenceRestLayer[]|>,
{NetPort["Input"]->"encoder"->NetPort["decoder","State"],
NetPort["Target"]->NetPort["decoder","Input"],
"decoder"->NetPort["loss","Input"],
NetPort["Target"]->"rest"->NetPort["loss","Target"]},
"Input"->inputEnc,"Target"->targetEnc]

Train the net (this procedure is often referred to as "teacher forcing"):

result=NetTrain[trainingNet,trainingData,All,ValidationSet->testData,MaxTrainingRounds->50]
trainedNet=result["TrainedNet"]

In this case of three-digit integer addition, there are only 1999 possible outputs. It is feasible to calculate the loss for each possible output and find the one that minimizes the loss:

predict[input_]:=First@Ordering[trainedNet[<|"Input"->ConstantArray[input,1999],"Target"->Table[ToString[i],{i,1999}]|>],1]

Predict the output, given a number of inputs and the trained net:

predict/@{"34+200","32+87","344+77"}

The loss for each output, given an input, can also be visualized:

input="200+300";
ListPlot[trainedNet[<|"Input"->ConstantArray[input,1999],"Target"->Table[IntegerString[i],{i,1999}]|>],AxesLabel->{"sum","loss"}]

A more efficient way of obtaining predictions is to generate the output until the (RefLink[EndOfString, paclet:ref/EndOfString]) virtual character is reached.

First, extract the trained "encoder" and "decoder" subnets from the trained (RefLink[NetGraph, paclet:ref/NetGraph]), and attach appropriate encoders and decoders:

trainedEncoder=NetReplacePart[NetExtract[trainedNet,"encoder"],"Input"->inputEnc]

trainedDecoder=NetExtract[trainedNet,"decoder"]

Use a (RefLink[SequenceLastLayer, paclet:ref/SequenceLastLayer]) to make a version of the decoder that only produces predictions for the last character of the answer, given the previous characters. Here a character decoder is attached for the probability vector using the same alphabet as the "Target" input:

targetDec=NetDecoder[{"Characters",{DigitCharacter,{StartOfString,EndOfString}->Automatic},"InputDepth"->1}];
charPredictor1=NetGraph[{trainedDecoder,SequenceLastLayer[]},{1->2},"Input"->targetEnc,"Output"->targetDec]

Apply the decoder to a partial answer:

state=trainedEncoder["25+25"];
charPredictor1[<|"Input"->"5","State"->state|>]//InputForm

When the decoder predicts the end of the target string, an empty string will be produced:

charPredictor1[<|"Input"->"50","State"->state|>]//InputForm

Now define a prediction function that takes the "encoder" and "decoder" nets and an input string. The function will compute successively longer results until the decoder claims to be finished:

predict2[input_]:=Module[
{encoded=trainedEncoder[input],result="",prediction},
Do[
prediction=charPredictor1[<|"Input"->result,"State"->encoded|>];
If[prediction==="",Break[]];
result=result<>prediction,
100]; (* stop at length 100, just in case *)
result
]

Evaluate this prediction function on input data:

predict2["564+200"]//InputForm

inputs={"564+200","44+87","344+67"};
predict2/@inputs

This is an good approximation to the first method, which finds the exact maximum likelihood answer:

predict/@inputs

The naive technique of generating by passing in each partial answer to the decoder net to derive the next character has time complexity of (n^2), where n is the length of the output sequence, and so is not appropriate for generating longer sequences. (RefLink[NetStateObject, paclet:ref/NetStateObject]) can be used to generate with time complexity of n.

First, a decoder is created that takes a single character and predicts the next character. The recurrent state of the (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer]) is handled by a (RefLink[NetStateObject, paclet:ref/NetStateObject]) at a later point.

Obtain the trained (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer]) and (RefLink[LinearLayer, paclet:ref/LinearLayer]) from the trained net:

gru=NetExtract[trainedNet,{"decoder",3}]
linear=NetExtract[trainedNet,{"decoder",4,"Net"}]

Define a "Class" encoder and decoder that will encode and decode individual characters, as well as the special classes to indicate the start and end of the string:

digits=Characters["0123456789"];
classEnc=NetEncoder[{"Class",Append[digits,StartOfString]}];
classDec=NetDecoder[{"Class",Append[digits,EndOfString]}];

Define a net that takes a single character, runs one step of the (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer]), and produces a single softmax prediction:

charPredictor2=NetChain[{UnitVectorLayer[],ReshapeLayer[{1,11}],gru,ReshapeLayer[{150}],linear,SoftmaxLayer[]},"Input"->classEnc,"Output"->classDec]

This predictor has an internal recurrent state, as revealed by (RefLink[NetInformation, paclet:ref/NetInformation]):

NetInformation[charPredictor2,"RecurrentStatesPositionList"]

Create a function that uses (RefLink[NetStateObject, paclet:ref/NetStateObject]) to memorize the internal recurrent state, which is seeded from the code produced by the trained encoder:

predict3[input_]:=Module[{code,sobj,res},
code=trainedEncoder[input];
sobj=NetStateObject[charPredictor2,<|{3,"State"}->code|>];
res=NestWhileList[sobj,StartOfString,#=!=EndOfString&];
StringJoin@res[[2;;-2]]
]

Apply the function to some inputs to obtain predicted sequences:

predict3["111+111"]//FullForm

predict3/@{"564+200","44+87","344+67"}

Obtain the accuracy of the trained net on the test set:

prediction=predict3/@Keys[testData];
accuracy=N@Mean@Boole@MapThread[Equal,{prediction,Values[testData]}]

Integer Sorting

In this example, a net is trained to sort lists of integers. For example, the net takes as input {3,2,4} and needs to return {2,3,4}. This example also demonstrates the use of a (RefLink[SequenceAttentionLayer, paclet:ref/SequenceAttentionLayer]), which significantly improves the performance of neural nets on many sequence learning tasks.

Generate a test and training set consisting of lists of integers between 1 and 6:

digits=Range[6];
seqs=RandomSample[Flatten[Table[Tuples[digits,n],{n,3,6}],1]];
data=Map[#->Sort[#]&,seqs];
{testData,trainData}=TakeDrop[data,Ceiling[Length[data]/10]];

Display three random samples drawn from the training set:

RandomSample[trainData,3]

Define a (RefLink[NetGraph, paclet:ref/NetGraph]) with an (RefLink[AttentionLayer, paclet:ref/AttentionLayer]):

net = NetGraph[<|
"enc"->{EmbeddingLayer[12],LongShortTermMemoryLayer[50]},"dec"->LongShortTermMemoryLayer[50],
"key"->NetMapOperator[50],
"value"->NetMapOperator[50],
"attend"->AttentionLayer["Dot"],
"cat"->CatenateLayer[2],"classify"->{NetMapOperator[LinearLayer[]],SoftmaxLayer[]}
|>,{
"enc"->"key"->NetPort["attend","Key"],"enc"->"value"->NetPort["attend","Value"],
"enc"->"dec"->NetPort["attend","Query"],
"dec"->"cat","attend"->"cat","cat"->"classify"},"Input"->{"Varying",NetEncoder[{"Class",digits}]},"Output"->{"Varying",NetDecoder[{"Class",digits}]}
]

Train the net:

result=NetTrain[net,trainData,All,MaxTrainingRounds->3,ValidationSet->testData]
trained=result["TrainedNet"]

Use the net to sort a list of integers:

trained[{6,6,1,4,2}]

Optical Character Recognition (OCR) on a Toy Dataset

The optical character recognition problem is to take an image containing a sequence of characters and return the list of characters. One simple approach is to preprocess the image to produce images containing only a single character and do classification. This is a fragile approach and completely fails for domains such as cursive handwriting, where the characters run together.

First, generate training and test data, which consists of images of words and the corresponding word string:

imgMap=\[LeftAssociation]" "->,"a"->,"b"->,"c"->,"d"->,"e"->,"f"->,"g"->,"h"->,"i"->,"j"->,"k"->,"l"->,"m"->,"n"->,"o"->,"p"->,"q"->,"r"->,"s"->,"t"->,"u"->,"v"->,"w"->,"x"->,"y"->,"z"->\[RightAssociation];

maxLen=7;
wordList=ToLowerCase[Select[WordList[],StringLength[#]≤maxLen&&LetterQ[#]&]];
SeedRandom[1234];
wordList=RandomSample[StringPadRight[wordList,maxLen]];
dataset=Dataset@Map[<|"Input"->ImageAssemble[Lookup[imgMap,Characters[#]]],"Output"->StringTrim[#]|>&,wordList];

Split the dataset into a test set and a training set:

{testData,trainData}=TakeDrop[dataset,Ceiling[Length[dataset]/10]];

Take a (RefLink[RandomSample, paclet:ref/RandomSample]) of the training set:

RandomSample[trainData,4]

The list of characters used:

chars=CharacterRange["a","z"]

The decoder is a beam search decoder with a beam size of 50:

decoder=NetDecoder[{"CTCBeamSearch",chars,"BeamSize"->50}]

Define a net that takes an image and then treats the width dimension as a sequence dimension. A sequence of probability vectors over the width dimension is produced:

ocrNet=NetChain[{ConvolutionLayer[20,3],BatchNormalizationLayer[],Ramp,PoolingLayer[2],ConvolutionLayer[15,3],BatchNormalizationLayer[],Ramp,PoolingLayer[2],FlattenLayer[1],TransposeLayer[],GatedRecurrentLayer[19],GatedRecurrentLayer[19],NetMapOperator[LinearLayer[Length[chars]+1]],SoftmaxLayer[]},"Input"->NetEncoder[{"Image",{63,15},"Grayscale"}],"Output"->decoder]

Define a (RefLink[CTCLossLayer, paclet:ref/CTCLossLayer]) with a character (RefLink[NetEncoder, paclet:ref/NetEncoder]) attached to the target port:

loss=CTCLossLayer["Target"->NetEncoder[{"Characters",chars}]]

Train the net using the CTC loss:

result=NetTrain[ocrNet,trainData,All,LossFunction->loss,MaxTrainingRounds->20,ValidationSet->testData]
trainedNet=result["TrainedNet"]

Evaluate the trained net on images from the test set:

testIms=Normal@testData[1;;3,"Input"]

trainedNet[testIms]

Obtain the top 5 decodings for an image, along with the negative log-likelihood of each decoding:

trainedNet[,{"TopNegativeLogLikelihoods",5}]

Question Answering

Simple RNN Trained on the bAbI QA Dataset

Train a question-answering net on the first task (Single Supporting Fact) of the bAbI QA dataset using a recurrent network.

First, obtain the training and validation data:

ro=ResourceObject["The 20-Task bAbI Question-Answering Dataset v1.2"]

traindata=ResourceData[ro,"TrainingData"]["Task1"];
testdata=ResourceData[ro,"TestData"]["Task1"];

traindata[[All,100]]

Obtain the list of classes used and dictionary used for the questions and contexts:

dict=Union@@TextWords[Join[traindata["Context"],traindata["Question"]]]

classes=Union[traindata["Answer"]]

Define a net that takes a question string and a context string and returns an answer:

enc=NetEncoder[{"Tokens",Join[dict,{"."}]}];
net=NetGraph[<|
"context"->{EmbeddingLayer[50],DropoutLayer[0.3]},"question"->{EmbeddingLayer[50],DropoutLayer[0.3],LongShortTermMemoryLayer[50],SequenceLastLayer[]},"cat"->CatenateLayer[],"classifier"->{LongShortTermMemoryLayer[50],SequenceLastLayer[],DropoutLayer[0.3],LinearLayer[Length[classes]],SoftmaxLayer[]}|>,
{NetPort["Context"]->"context",NetPort["Question"]->"question",
{"question","context"}->"cat","cat"->"classifier"->NetPort["Answer"]},
"Context"->enc,"Question"->enc,"Answer"->NetDecoder[{"Class",classes}]]

Train the network for three training rounds. (RefLink[NetTrain, paclet:ref/NetTrain]) will automatically attach a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) using the same classes that were provided to the decoder:

result=NetTrain[net,traindata,All,ValidationSet->testdata,MaxTrainingRounds->30]
trainedNet=result["TrainedNet"]

Make a prediction with the trained net:

trainedNet[\[LeftAssociation]"Context"->"Sandra went to the office. Sandra travelled to the bathroom. Mary went back to the bedroom. Daniel moved to the garden. John journeyed to the garden. Daniel went back to the hallway. John journeyed to the bedroom. Mary travelled to the kitchen. Sandra went to the bedroom. Daniel went to the garden.","Question"->"Where is Daniel?"\[RightAssociation]]

Obtain the accuracy of the trained net on the test set:

N@Count[trainedNet[KeyDrop[testdata,"Answer"]]-testdata["Answer"],0]/Length[testdata["Answer"]]

Memory Network Trained on the bAbI QA Dataset

Train a question-answering net on the first task (Single Supporting Fact) of the bAbI QA dataset, using a memory network based on Sukhbaatar et al., "End-to-End Memory Networks", 2015.

First, obtain the training and validation data:

ro=ResourceObject["The 20-Task bAbI Question-Answering Dataset v1.2"]

traindata=ResourceData[ro,"TrainingData"]["Task1"];
testdata=ResourceData[ro,"TestData"]["Task1"];

traindata[[All,100]]

The memory net has layers (such as (RefLink[TransposeLayer, paclet:ref/TransposeLayer])) that currently do not support variable-length sequences.

Convert all strings to lists of tokens and use left padding (which has better performance than right padding for this example) to ensure these lists are the same length:

maxContextLen=68;
maxQueryLen=4;
stringPreprocess[strings_]:=PadLeft[#,maxContextLen,Padding]&/@StringSplit[strings,{" ",", "->",",". "->".","?"->"?","."->"."}];

traindata["Context"]=stringPreprocess[traindata["Context"]];
testdata["Context"]=stringPreprocess[testdata["Context"]];
traindata["Question"]=StringSplit[traindata["Question"],{"?"->"?"," "}];
testdata["Question"]=StringSplit[testdata["Question"],{"?"->"?"," "}];

The "Context" input is now a list of length 68 padded with the (RefLink[Padding, paclet:ref/Padding]) symbol:

traindata[[All,3]]

Obtain the list of classes used and dictionary used for the questions and contexts:

dict=Union@Flatten[Join[traindata["Context"],traindata["Question"]]]

classes=Union[traindata["Answer"]]

Define the net:

embedDim=64;
enc=NetEncoder[{"Class",dict}];
memNet=NetGraph[<|
"context1"->{EmbeddingLayer[embedDim],DropoutLayer[0.3]},"context2"->{EmbeddingLayer[maxQueryLen],DropoutLayer[0.3]},"question"->{EmbeddingLayer[embedDim],DropoutLayer[0.3]},"trans1"->TransposeLayer[],"trans2"->TransposeLayer[],
"featureMatch"->DotLayer[],"softmax"->SoftmaxLayer[],
"featureCat"->CatenateLayer[2],"response"->TotalLayer[],"classify"->{LongShortTermMemoryLayer[32],SequenceLastLayer[],DropoutLayer[0.3],LinearLayer[],SoftmaxLayer[]}|>,{
NetPort["Question"]->"question",
NetPort["Context"]->"context1",
NetPort["Context"]->"context2",
"question"->"trans1",
{"context1","trans1"}->"featureMatch"->"softmax",
{"softmax","context2"}->"response"->"trans2",
{"trans2","question"}->"featureCat"->"classify"->NetPort["Answer"]
},"Context"->{maxContextLen,enc},"Question"->{maxQueryLen,enc},"Answer"->NetDecoder[{"Class",classes}]];

Train the net using the "RMSProp" optimization method, which improves learning performance for this example:

result=NetTrain[memNet,traindata,All,MaxTrainingRounds-> 65,ValidationSet->testdata,Method->"RMSProp",BatchSize->32]
trainedNet=result["TrainedNet"]

Obtain the accuracy of the net on the test data:

N@Count[trainedNet[KeyDrop[testdata,"Answer"]]-testdata["Answer"],0]/Length[testdata["Answer"]]

Language Modeling

Character-Level Language Model

Training an English character-level language model.

First, create 300,000 training examples of 25 characters each from two novels:

corpus=StringJoin[ResourceData["The American"]," " , ResourceData["Don Quixote - English"]];

sampleCorpus[corpus_,len_,num_]:=Module[
{positions=RandomInteger[{len+1,StringLength[corpus]-1},num]},
Thread[StringTake[corpus,{#-len,#}&/@positions]->StringPart[corpus,positions+1]]];
trainingData=sampleCorpus[corpus,25,300000];

The data is of the form of a classification problem: given a sequence of characters, predict the next one. A sample of the data:

RandomSample[trainingData,5]//Map[InputForm]//TableForm

Obtain the list of all characters in the text:

characters=Union[Characters[corpus]]

Define a net that takes in a string of characters and returns a prediction for the next character:

net=NetChain[{UnitVectorLayer[],GatedRecurrentLayer[128],GatedRecurrentLayer[128],SequenceLastLayer[],LinearLayer[],SoftmaxLayer[]},
"Input"->NetEncoder[{"Characters",characters}],
"Output"->NetDecoder[{"Class",characters}]]

Train the net. This can take up to an hour on a CPU, so use (RefLink[TargetDevice, paclet:ref/TargetDevice])->"GPU" if you have an NVIDIA graphics card available. A modern GPU should be able to complete this example in about 7 minutes:

results=NetTrain[net,trainingData,All,ValidationSet->Scaled[0.1],BatchSize->64,MaxTrainingRounds->10,TargetDevice->"CPU"]

trainedNet=results["TrainedNet"]

Predict the next character, given a sequence of characters:

trainedNet["Why are th"]

Generate 100 characters of text, given a start text. Note that this method uses (RefLink[NetStateObject, paclet:ref/NetStateObject]) to efficiently generate long sequences of text:

generate[net_,start_,len_]:=StringJoin@NestList[NetStateObject[net],start,len]

generate[trainedNet,"Why are w",100]

You can get more interesting text by sampling from the probability distribution of predictions:

generateSample[net_,start_,len_]:=Block[{obj=NetStateObject[net]},
StringJoin@NestList[obj[#,"RandomSample"]&,start,len]]

generateSample[trainedNet,"Why are w",100]

An alternative and equivalent formulation of this learning problem, requiring only a single string as input, is to separate the last character from the rest of the sequence inside the net.

Use (RefLink[SequenceMostLayer, paclet:ref/SequenceMostLayer]) and (RefLink[SequenceLastLayer, paclet:ref/SequenceLastLayer]) in a graph to separate the last character from the rest:

netAlt=NetGraph[
<|"rnn"->net,"most"->SequenceMostLayer[],"last"->SequenceLastLayer[],"loss"->CrossEntropyLossLayer["Index"]|>,
{NetPort["Input"]->"most"->"rnn"->"loss",
NetPort["Input"]->"last"->NetPort["loss","Target"]},
"Input"->NetEncoder[{"Characters",characters}]]

Train this net on the input sequences from the original training data (technically, this means you end up training the net on slightly shorter sequences):

result=NetTrain[netAlt,<|"Input"->Keys[trainingData]|>, All,ValidationSet->Scaled[0.1],BatchSize->64,MaxTrainingRounds->10]
netAltTrained=result["TrainedNet"]

A more efficient way to train language models is to use "teacher forcing", in which the net simultaneously predicts the entire sequence, rather than just the last letter.

First, build the net that does prediction of an entire sequence at once. This differs from the previous prediction nets in that the (RefLink[LinearLayer, paclet:ref/LinearLayer]) is mapped and a matrix softmax is performed, instead of taking the last element and doing an ordinary vector softmax:

n=Length[characters];
predict=NetChain[{UnitVectorLayer[n],GatedRecurrentLayer[128],GatedRecurrentLayer[128],NetMapOperator[LinearLayer[n]],SoftmaxLayer[]}]

Now build the forcing network, which takes a target sentence and presents it to the network in a "staggered" fashion: for a length-26 sentence, present characters 1 through 25 to the net so that it produces predictions for characters 2 through 26, which are compared with the real characters via the (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) to produce a loss:

teacherForcingNet=NetGraph[
<|"predict"->predict,"rest"->SequenceRestLayer[],"most"->SequenceMostLayer[],"loss"->CrossEntropyLossLayer["Index"]|>,
{NetPort["Input"]->"most"->"predict"->NetPort["loss","Input"],NetPort["Input"]->"rest"->NetPort["loss","Target"]},
"Input"->NetEncoder[{"Characters",characters}]]

Train the net on the input sequences from the original data. On a typical CPU, this should take around 15 minutes, compared to around 2 minutes on a modern GPU. As teacher forcing is a more efficient technique, you can afford to use a smaller number of training rounds:

result=NetTrain[teacherForcingNet,<|"Input"->Keys[trainingData]|>,All,BatchSize->64,MaxTrainingRounds->5,TargetDevice->"CPU",ValidationSet->Scaled[0.1]]

Extract the prediction chain from the results:

trained=NetExtract[result["TrainedNet"],"predict"]

Build a single-character prediction chain from it:

predictNet=NetJoin[
NetTake[trained,3,"Input"->Automatic],
{SequenceLastLayer[],NetExtract[trained,{4,"Net"}],SoftmaxLayer[]},
"Input"->NetEncoder[{"Characters",characters}],
"Output"->NetDecoder[{"Class",characters}]]

Test the predictor:

predictNet["Why are th"]

Generate 100 characters of text, given a start text. Note that this method uses (RefLink[NetStateObject, paclet:ref/NetStateObject]) to efficiently generate long sequences of text:

generate[predictNet,"It was ",100]

You can get more interesting text by sampling from the probability distribution of predictions:

generateSample[predictNet,"Why are w",100]