({{, (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer])[f] 
represents a net layer that applies a unary function f to every element of the input array.}, {, (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer])["name"]
applies the function specified by "name".}})

OpenersNotesSection(Spacer1)Details and Options

The function f can be any one of the following: (RefLink[Ramp, paclet:ref/Ramp]), (RefLink[LogisticSigmoid, paclet:ref/LogisticSigmoid]), (RefLink[Tanh, paclet:ref/Tanh]), (RefLink[ArcTan, paclet:ref/ArcTan]), (RefLink[ArcTanh, paclet:ref/ArcTanh]), (RefLink[Sin, paclet:ref/Sin]), (RefLink[Sinh, paclet:ref/Sinh]), (RefLink[ArcSin, paclet:ref/ArcSin]), (RefLink[ArcSinh, paclet:ref/ArcSinh]), (RefLink[Cos, paclet:ref/Cos]), (RefLink[Cosh, paclet:ref/Cosh]), (RefLink[ArcCos, paclet:ref/ArcCos]), (RefLink[ArcCosh, paclet:ref/ArcCosh]), (RefLink[Log, paclet:ref/Log]), (RefLink[Exp, paclet:ref/Exp]), (RefLink[Sqrt, paclet:ref/Sqrt]), (RefLink[Abs, paclet:ref/Abs]), (RefLink[Gamma, paclet:ref/Gamma]), (RefLink[LogGamma, paclet:ref/LogGamma]).

In general, f can be any object that when applied to a single argument gives any combination of (RefLink[Ramp, paclet:ref/Ramp]), (RefLink[LogisticSigmoid, paclet:ref/LogisticSigmoid]), etc., together with (RefLink[Plus, paclet:ref/Plus]), (RefLink[Subtract, paclet:ref/Subtract]), (RefLink[Times, paclet:ref/Times]), (RefLink[Divide, paclet:ref/Divide]), (RefLink[Power, paclet:ref/Power]), (RefLink[Min, paclet:ref/Min]), (RefLink[Max, paclet:ref/Max]), (RefLink[Clip, paclet:ref/Clip]), and numbers.

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) supports the following values for "name":

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer])[…][input] explicitly computes the output from applying the layer.

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer])[…][{(Subscript[input, 1]),(Subscript[input, 2]),…}] explicitly computes outputs for each of the (Subscript[input, i]).

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) is typically used inside (RefLink[NetChain, paclet:ref/NetChain]), (RefLink[NetGraph, paclet:ref/NetGraph]), etc.

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) exposes the following ports for use in (RefLink[NetGraph, paclet:ref/NetGraph]) etc.:

When it cannot be inferred from other layers in a larger net, the option "Input"->{(Subscript[n, 1]),(Subscript[n, 2]),…} can be used to fix the input dimensions of (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]).

OpenersPrimaryExamplesSection(Spacer1)Examples  (12)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that computes (RefLink[Tanh, paclet:ref/Tanh]) of each input element:

ElementwiseLayer[Tanh]

Create an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that multiplies its input by a fixed constant:

elem=ElementwiseLayer[#*3&]

Apply the layer to an input vector:

elem[{1,2,3}]

OpenersExampleSection1(Spacer1)Scope  (3)

Create a (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that computes a "hard sigmoid":

hsig=ElementwiseLayer[Clip[(#+1)/2,{0,1}]&,"Input"->"Real"]

Apply the layer to a batch of inputs:

hsig[{-1,0,0.5,1,1.5}]

Plot the behavior of the layer:

Plot[hsig[x],{x,-2,2}]

Create an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that takes vectors of size 3:

elem=ElementwiseLayer[Ramp,"Input"->3]

Apply the layer to a single vector:

elem[{1,2,3}]

When applied, the layer will automatically thread over a batch of vectors:

elem[{{1,2,3},{-1,-2,-3}}]

Create an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that computes a Gaussian:

elem=ElementwiseLayer[Exp[-#^2]&]

Apply the layer to a range of values and plot the result:

ListPlot[elem[Range[-2,2,.1]]]

OpenersExampleSection3(Spacer1)Applications  (3)

Train a 16-layer-deep, self-normalizing network described in "Self-Normalizing Neural Networks", G. Klambauer et. al., 2017, on the UCI Letter dataset. Obtain the data:

train=ExampleData[{"MachineLearning","UCILetter"},"TrainingData"];
test=ExampleData[{"MachineLearning","UCILetter"},"TestData"];

RandomSample[train,2]

Self-normalizing nets assume that the input data has mean of 0 and variance of 1. Standardize the test and training data:

extractor=FeatureExtraction[N@Keys[train],"StandardizedVector"];
trainN=extractor[Keys[train]]->Values[train];
testN=extractor[Keys[test]]->Values[test];

Verify that the sample mean and variance of the training data is 1 and 0, respectively:

Mean@First[trainN]

Variance@First[trainN]

Define a 16-layer, self-normalizing net with "AlphaDropout":

module[rate_]:=NetChain[{LinearLayer[50],ElementwiseLayer["SELU"],DropoutLayer[rate,"Method"->"AlphaDropout"]}]

net=NetChain[Join[Table[module[0.01],15],{LinearLayer[],SoftmaxLayer[]}]]

Train the net:

netT=NetTrain[net,trainN,LossFunction->CrossEntropyLossLayer["Index","Target"->NetEncoder[{"Class",Union@Values[train]}]],ValidationSet->testN,MaxTrainingRounds->150,Method->"SGD"]

Obtain the accuracy:

NetMeasurements[netT,testN,"Accuracy"]

Compare the accuracy against the "RandomForest" method in (RefLink[Classify, paclet:ref/Classify]):

c=Classify[trainN,Method->"RandomForest",PerformanceGoal->"Quality"]

ClassifierMeasurements[c,testN,"Accuracy"]

Allow a classification network to deal with a "non-separable" problem. Create a synthetic training set consisting of points on a disk, separated into two classes by the circle r=0.5.

points=RandomPoint[Disk[],1000];
labels=Thread[Map[Norm,points]<0.5];
trainingData=points->labels;

ListPlot[{Pick[points,labels,True],Pick[points,labels,False]},AspectRatio->1]

Create a layer composed of two (RefLink[LinearLayer, paclet:ref/LinearLayer]) layers, and a final transformation into a probability using an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]):

net=NetChain[{LinearLayer[20],LinearLayer[],ElementwiseLayer[LogisticSigmoid]},"Output"->NetDecoder["Boolean"]]

Train the network on the data:

trained=NetTrain[net,trainingData];

The net was not able to separate the two classes:

ContourPlot[trained[{x,y},None],{x,-1,1},{y,-1,1}]

Because (RefLink[LinearLayer, paclet:ref/LinearLayer]) is an affine layer, stacking the two layers without an intervening nonlinearity is equivalent to using a single layer. A single line in the plane cannot separate the two classes, which is the level set of a single (RefLink[LinearLayer, paclet:ref/LinearLayer]).

Train a similar net that has a (RefLink[Ramp, paclet:ref/Ramp]) nonlinearity between the two layers:

net=NetChain[{LinearLayer[20],ElementwiseLayer[Tanh],LinearLayer[],ElementwiseLayer[LogisticSigmoid]},"Output"->NetDecoder["Boolean"]]

The net can now separate the classes:

trained=NetTrain[net,trainingData];
ContourPlot[trained[{x,y},None],{x,-1,1},{y,-1,1}]

Binary classification tasks require that the output of a net be a probability. (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer])[(RefLink[LogisticSigmoid, paclet:ref/LogisticSigmoid])] can be used to take an arbitrary scalar and produce a value between 0 and 1. Create a net that takes a vector of length 2 and produces a binary prediction:

net=NetChain[{LinearLayer[],ElementwiseLayer[LogisticSigmoid]},"Output"->NetDecoder["Boolean"]]

Train the net to decide if the first number in the vector is greater than the second:

trained=NetTrain[net,Flatten@Table[{i,j}->(i>j),{i,5},{j,5}]]

Evaluate the net on several inputs:

trained[{3,5}]

trained[{5,3}]

The underlying output is a probability, which can be seen by disabling the "Boolean" decoder:

trained[{4,4.01},None]

OpenersExampleSection4(Spacer1)Properties & Relations  (1)

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) is automatically used when an appropriate function is specified in a (RefLink[NetChain, paclet:ref/NetChain]) or (RefLink[NetGraph, paclet:ref/NetGraph]):

chain=NetChain[{LinearLayer[3],Ramp,LinearLayer[5]}]

NetExtract[chain,2]

OpenersExampleSection5(Spacer1)Possible Issues  (3)

(RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) cannot accept symbolic inputs:

ElementwiseLayer[Ramp][{b,2,3}]

Certain choices of f can produce failures for inputs outside their domain:

ElementwiseLayer[Sqrt][{-1}]

Certain functions are not supported directly:

ElementwiseLayer[Zeta]

Approximate the (RefLink[Zeta, paclet:ref/Zeta]) function using a rational function:

approxZeta=Function[x,Evaluate@N@PadeApproximant[Zeta[x],{x,1,3}]]

The approximation is good over the range (-10,10):

Plot[{approxZeta[x],Zeta[x]},{x,-10,10}]

Construct an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) using the approximation:

elem=ElementwiseLayer[approxZeta]

Measure the error of the approximation on specific inputs:

inputs=N[{-1,0,2,3,4}];
elem[inputs]-Zeta[inputs]

The network will fail when evaluated at a pole:

Zeta[1]

elem[{1}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[ThreadingLayer, paclet:ref/ThreadingLayer])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[SoftmaxLayer, paclet:ref/SoftmaxLayer])  ▪  (RefLink[DropoutLayer, paclet:ref/DropoutLayer])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})