({{, (RefLink[NetDecoder, paclet:ref/NetDecoder])["Tokens"] 
represents a decoder that converts a sequence of probability vectors to a string of English vocabulary words.}, {, (RefLink[NetDecoder, paclet:ref/NetDecoder])[{"Tokens","language"}]
represents a decoder that uses a standard vocabulary for the given language.}, {, (RefLink[NetDecoder, paclet:ref/NetDecoder])[{"Tokens",{"(Subscript[token, 1])","(Subscript[token, 2])",…}}]
represents a decoder that uses a specified list of tokens as the vocabulary.}, {, (RefLink[NetDecoder, paclet:ref/NetDecoder])[{"Tokens",…,"param"->val}]
represents a decoder in which additional parameters have been specified.}})

OpenersNotesSection(Spacer1)Details

(RefLink[NetDecoder, paclet:ref/NetDecoder])[…][input] applies the decoder to an input to produce an output.

(RefLink[NetDecoder, paclet:ref/NetDecoder])[…][{(Subscript[input, 1]),(Subscript[input, 2]),…}] applies the decoder to a list of inputs to produce a list of outputs.

The input to the decoder input is either a vector of probabilities or a sequence of probability vectors. Each probability vector sums to 1. The length of each probability vector is the size of the vocabulary.

(RefLink[NetDecoder, paclet:ref/NetDecoder])["Tokens"] is equivalent to (RefLink[NetDecoder, paclet:ref/NetDecoder])[{"Tokens","English"}].

A decoder can be attached to an output port of a net by specifying "port"->(RefLink[NetDecoder, paclet:ref/NetDecoder])[…] when constructing the net.

With the parameter "IgnoreCase"->(RefLink[True, paclet:ref/True]), tokens are converted to lowercase after decoding. The default is (RefLink[True, paclet:ref/True]).

(RefLink[NetDecoder, paclet:ref/NetDecoder])[…][data,prop] can be used to calculate a specific property for the input data.

When a "Tokens" decoder is attached to a net, net[data,prop] or net[data,"oport"->prop] can be used to calculate a specific property of the decoded output.

The "Tokens" decoder only supports the bypass property. Setting prop to (RefLink[None, paclet:ref/None]) bypasses decoding and returns the input to the decoder.

OpenersPrimaryExamplesSection(Spacer1)Examples  (3)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create a token decoder for English text:

dec=NetDecoder["Tokens"]

Decode an English sentence:

dec[UnitVector[40236,#]&/@{16146,39956}]

Decode a nonsense sentence from a random matrix of probabilities:

dec@RandomReal[1,{5,40236}]

Create a decoder with two tokens:

dec=NetDecoder[{"Tokens",{"hello","bye"}}]

Evaluate the decoder on a sequence of probability vectors:

dec[{{0.1,0.85,0.05},{0.5,0.4,0.1},{0.3,0.5,0.2}}]

OpenersExampleSection1(Spacer1)Scope  (1)

Attach a token decoder to the output port of a net:

net=NetInitialize@NetMapOperator[LinearLayer[],"Output"->NetDecoder["Tokens"],"Input"->{2,3}]

Evaluating the net on a sequence of probability vectors returns a string:

net[{{0.2,0.3,0.2},{-0.4,2,4}}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[NetDecoder, paclet:ref/NetDecoder])  ▪  (RefLink[NetEncoder, paclet:ref/NetEncoder])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])}, {Related Net Encoders:  (RefLink[Tokens, paclet:ref/netencoder/Tokens])  ▪  (RefLink[Characters, paclet:ref/netencoder/Characters])  ▪  (RefLink[Class, paclet:ref/netencoder/Class])  ▪  (RefLink[BPESubwordTokens, paclet:ref/netencoder/BPESubwordTokens])}, {Related Net Decoders:  (RefLink[Characters, paclet:ref/netdecoder/Characters])  ▪  (RefLink[Class, paclet:ref/netdecoder/Class])  ▪  (RefLink[BPESubwordTokens, paclet:ref/netdecoder/BPESubwordTokens])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})