({{, (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer])[{h,w}] 
represents a net layer that applies an affine transformation to an input of size c×(Subscript[h, 0])×(Subscript[w, 0]) and returns an output of size c×h×w.}})

OpenersNotesSection(Spacer1)Details and Options

(RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) exposes the following ports for use in (RefLink[NetGraph, paclet:ref/NetGraph]) etc.:

(RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer])[…][<|"Input"->in,"Parameters"->param|>] explicitly computes the output from applying the layer.

(RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer])[…][<|"Input"->{(Subscript[in, 1]),(Subscript[in, 2]),…},"Parameters"->{(Subscript[param, 1]),(Subscript[param, 2]),…}|>] explicitly computes output  for each of the (Subscript[in, i]) and (Subscript[param, i]).

When given a (RefLink[NumericArray, paclet:ref/NumericArray]) as input, the output will be a (RefLink[NumericArray, paclet:ref/NumericArray]).

(RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) is typically used inside (RefLink[NetGraph, paclet:ref/NetGraph]) to focus the attention of a later convolutional network on the best part of the image to perform a specific task.

When it cannot be inferred from other layers in a larger net, the option "Input"->{(Subscript[d, 1]),(Subscript[d, 2]),(Subscript[d, 3])} can be used to fix the input dimensions of (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]).

The six components of the vector provided to the port "Parameters", {(Subscript[z, h]),(Subscript[s, h]),(Subscript[t, h]),(Subscript[s, v]),(Subscript[z, v]),(Subscript[t, v])}, represent the parameters in the affine transformation matrix, where (Subscript[z, i]) represents zoom, (Subscript[s, i]) skewness and (Subscript[t, i]) translation, and the subscripts h and v indicate horizontal and vertical. The identity transformation is obtained when "Parameters" is {1,0,0,0,1,0}.

OpenersPrimaryExamplesSection(Spacer1)Examples  (5)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Create a (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) with output size 30×30:

SpatialTransformationLayer[{30,30}]

Create a (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) that expects an input of size 1×3×3 and returns an output of size 1×2×2:

transform=SpatialTransformationLayer[{2,2},"Input"->{1,3,3}]

Apply the layer to an input:

transform[<|"Parameters"->{0.5,0,0,0,0.5,0},"Input"->{{{1,0,0},{0,0,0},{0,0,1}}}|>]

OpenersExampleSection1(Spacer1)Scope  (1)

Create a (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) whose input is an image and whose output is an image:

spatial=SpatialTransformationLayer[{64,64},"Input"-> NetEncoder["Image"],"Output"-> NetDecoder["Image"]]

Apply the (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) to an image with a factor-2 zoom transformation:

img=;

spatial[<|"Input"-> img,"Parameters"->{0.5,0,0,0,0.5,0}|>]

Apply the (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]) using a sequence of zooms:

spatial[<|"Input"->Table[img,10],"Parameters"->Table[{i,0,0,0,i,0},{i,0.2,2,1/5}]|>]

OpenersExampleSection3(Spacer1)Applications  (1)

Train a digit recognizer on the MNIST database of handwritten digits using a convolutional neural network with a (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]). First obtain the training and test data:

resource = ResourceObject["MNIST"];
trainingData=ResourceData[resource,"TrainingData"];
testData=ResourceData[resource,"TestData"];

Define a function to apply extra padding and random translations to the training and test data:

f[image_]:=ImagePerspectiveTransformation[ImagePad[ColorNegate[image],5],TranslationTransform[RandomReal[{-0.2,0.2},2]]];

f/@{,,}

Create new training and test data using the function (this should take about a minute):

{trainingData2,testData2}=MapAt[f,{trainingData,testData},{All,All,1}];

RandomSample[trainingData2,6]

Create a network that uses the image to predict the best affine transformation to apply to the image to extract the digit:

localizer=NetChain[{PoolingLayer[4,4],FlattenLayer[],LinearLayer[6]}]

Create a convolutional classification net to use the subimage extracted by the localization net:

classifier=NetChain[{
ConvolutionLayer[16,4],BatchNormalizationLayer[],Ramp,PoolingLayer[4],ConvolutionLayer[32,4],BatchNormalizationLayer[],Ramp,PoolingLayer[4],LinearLayer[10],SoftmaxLayer[]},
"Output"->NetDecoder[{"Class",Range[0,9]}],
"Input"->{1,16,16}]

Attach the classification network and the localization network to a spatial transformation layer:

net=NetGraph[<|
"localizer"->localizer,"transformer"->SpatialTransformationLayer[{16,16}],"classifier"->classifier|>,
{"localizer"->NetPort["transformer","Parameters"],"transformer"->"classifier"},
"Input" -> NetEncoder[{"Image",{38,38}, "Grayscale"}]]

Train the network:

trained=NetTrain[net,trainingData2,ValidationSet->testData2, MaxTrainingRounds->6]

If the classification network is removed, the effect of the spatial transformer can be visualized:

spatial=NetTake[trained,{NetPort["Input"], "transformer"}];
spatial=NetReplacePart[spatial,"Output"-> NetDecoder[{"Image","Grayscale"}]]

Apply the spatial transformer to some images from the validation set:

sample=Keys@RandomSample[testData2,5]

spatial[sample]

Obtain the accuracy of the network on the validation set:

NetMeasurements[trained,testData2,"Accuracy"]

OpenersExampleSection4(Spacer1)Properties & Relations  (1)

Apply an (RefLink[AffineTransform, paclet:ref/AffineTransform]) to the coordinates of an image using (RefLink[ImageTransformation, paclet:ref/ImageTransformation]):

img=;

m={{0.3,0.2},{-0.3,0.3}};
v={-0.02,0};
t=AffineTransform[{m,v}];
ImageTransformation[img,t,DataRange->{{-1,1},{-1,1}}]

Construct an equivalent set of parameters for (RefLink[SpatialTransformationLayer, paclet:ref/SpatialTransformationLayer]):

st=SpatialTransformationLayer[{128,128},"Input"->NetEncoder["Image"],"Output"->NetDecoder["Image"]]

makeTrans[{{m11_,m12_},{m21_,m22_}},{v1_,v2_}]:={m11,-m12,-v1,-m21,m22,v2};

st[<|"Input"->img,"Parameters"->makeTrans[m,v]|>]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer])  ▪  (RefLink[PoolingLayer, paclet:ref/PoolingLayer])  ▪  (RefLink[ResizeLayer, paclet:ref/ResizeLayer])  ▪  (RefLink[ImageAugmentationLayer, paclet:ref/ImageAugmentationLayer])  ▪  (RefLink[NetChain, paclet:ref/NetChain])  ▪  (RefLink[NetGraph, paclet:ref/NetGraph])  ▪  (RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[AffineTransform, paclet:ref/AffineTransform])  ▪  (RefLink[ImageResize, paclet:ref/ImageResize])  ▪  (RefLink[ImageTransformation, paclet:ref/ImageTransformation])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})