A softmax layer applies a softmax function to the input.

For classification problems, a softmax layer and then a classification layer must follow the final fully connected layer.

The output unit activation function is the softmax function:

where $$0\le y_r\le 1$$ and $${\displaystyle \sum _{j=1}^k{y}_j=1}$$.

The softmax function is the output unit activation function after the last fully connected layer for multi-class classification problems:

where $$0\le P\left(c_r|x,\theta \right)\le 1$$ and $${\displaystyle \sum _{j=1}^{k}P\left(c_j|x,\theta \right)=1}$$. Moreover, $$a_r=\ln \left(P\left(x,\theta |c_r\right)P\left(c_r\right)\right)$$, $$P\left(x,\theta |c_r\right)$$ is the conditional probability of the sample given class r, and $$P\left(c_r\right)$$ is the class prior probability.

The softmax function is also known as the normalized exponential and can be considered the multi-class generalization of the logistic sigmoid function [1].