Thesoftmax function, also known assoftargmax^[1]: 184 ornormalized exponential function,^[2]: 198 converts a vector ofK real numbers into aprobability distribution ofK possible outcomes. It is a generalization of thelogistic function to multiple dimensions, and is used inmultinomial logistic regression. The softmax function is often used as the lastactivation function of aneural network to normalize the output of a network to aprobability distribution over predicted output classes.

The softmax function takes as input a vectorz ofK real numbers, and normalizes it into aprobability distribution consisting ofK probabilities proportional to the exponentials of the input numbers. That is, prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in theinterval${\displaystyle (0,1)}$, and the components will add up to 1, so that they can be interpreted as probabilities. Furthermore, the larger input components will correspond to larger probabilities.

Formally, the standard (unit) softmax function${\displaystyle \sigma :{\mathbb{R}}^K\to (0,1{)}^K}$, where${\displaystyle K>1}$, takes a vector${\displaystyle \mathbf{z}=(z_1,\dots ,z_K)\in {\mathbb{R}}^K}$ and computes each component of vector${\displaystyle \sigma (\mathbf{z})\in (0,1{)}^K}$ with

$${\displaystyle \sigma (\mathbf{z}{)}_i=\frac{e^{z_i}}{\sum _{j=1}^K{e}^{z_j}}.}$$

In words, the softmax applies the standardexponential function to each element${\displaystyle z_i}$ of the input vector${\displaystyle \mathbf{z}}$ (consisting of${\displaystyle K}$ real numbers), and normalizes these values by dividing by the sum of all these exponentials. The normalization ensures that the sum of the components of the output vector${\displaystyle \sigma (\mathbf{z})}$ is 1. The term "softmax" derives from the amplifying effects of the exponential on any maxima in the input vector. For example, the standard softmax of${\displaystyle (1,2,8)}$ is approximately${\displaystyle (0.001,0.002,0.997)}$, which amounts to assigning almost all of the total unit weight in the result to the position of the vector's maximal element (of 8).

In general, instead ofe a differentbaseb > 0 can be used. As above, ifb > 1 then larger input components will result in larger output probabilities, and increasing the value ofb will create probability distributions that are more concentrated around the positions of the largest input values. Conversely, if0 < b < 1 then smaller input components will result in larger output probabilities, and decreasing the value ofb will create probability distributions that are more concentrated around the positions of the smallest input values. Writing${\displaystyle b=e^{\beta }}$ or${\displaystyle b=e^{-\beta }}$^[a] (for realβ)^[b] yields the expressions:^[c]

$${\displaystyle \sigma (\mathbf{z}{)}_i=\frac{e^{\beta z_i}}{\sum _{j=1}^K{e}^{\beta z_j}}\text{ or }\sigma (\mathbf{z}{)}_i=\frac{e^{-\beta z_i}}{\sum _{j=1}^K{e}^{-\beta z_j}}\text{ for }i=1,\dots ,K.}$$

A value proportional to the reciprocal ofβ is sometimes referred to as thetemperature:$\beta =1/kT$, wherek is typically 1 or theBoltzmann constant andT is the temperature. A higher temperature results in a more uniform output distribution (i.e. with higherentropy; it is "more random"), while a lower temperature results in a sharper output distribution, with one value dominating.

In some fields, the base is fixed, corresponding to a fixed scale,^[d] while in others the parameterβ (orT) is varied.

The Softmax function is a smooth approximation to thearg max function: the function whose value is theindex of a vector's largest element. The name "softmax" may be misleading. Softmax is not asmooth maximum (that is, asmooth approximation to themaximum function). The term "softmax" is also used for the closely relatedLogSumExp function, which is a smooth maximum. For this reason, some prefer the more accurate term "softargmax", though the term "softmax" is conventional in machine learning.^[3][4] This section uses the term "softargmax" for clarity.

Formally, instead of considering the arg max as a function with categorical output${\displaystyle 1,\dots ,n}$ (corresponding to the index), consider the arg max function withone-hot representation of the output (assuming there is a unique maximum arg):$${\displaystyle \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(z_1,\dots ,z_n)=(y_1,\dots ,y_n)=(0,\dots ,0,1,0,\dots ,0),}$$where the output coordinate${\displaystyle y_i=1}$ if and only if${\displaystyle i}$ is the arg max of${\displaystyle (z_1,\dots ,z_n)}$, meaning${\displaystyle z_i}$ is the unique maximum value of${\displaystyle (z_1,\dots ,z_n)}$. For example, in this encoding${\displaystyle \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(1,5,10)=(0,0,1),}$ since the third argument is the maximum.

This can be generalized to multiple arg max values (multiple equal${\displaystyle z_i}$ being the maximum) by dividing the 1 between all max args; formally1/k wherek is the number of arguments assuming the maximum. For example,${\displaystyle \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(1,5,5)=(0,1/2,1/2),}$ since the second and third argument are both the maximum. In case all arguments are equal, this is simply${\displaystyle \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(z,\dots ,z)=(1/n,\dots ,1/n).}$ Pointsz with multiple arg max values aresingular points (or singularities, and form the singular set) – these are the points where arg max is discontinuous (with ajump discontinuity) – while points with a single arg max are known as non-singular or regular points.

With the last expression given in the introduction, softargmax is now a smooth approximation of arg max: as⁠${\displaystyle \beta \to \mathrm{\infty }}$⁠, softargmax converges to arg max. There are various notions of convergence of a function; softargmax converges to arg maxpointwise, meaning for each fixed inputz as⁠${\displaystyle \beta \to \mathrm{\infty }}$⁠,${\displaystyle {\sigma }_{\beta }(\mathbf{z})\to \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(\mathbf{z}).}$ However, softargmax does notconverge uniformly to arg max, meaning intuitively that different points converge at different rates, and may converge arbitrarily slowly. In fact, softargmax is continuous, but arg max is not continuous at the singular set where two coordinates are equal, while the uniform limit of continuous functions is continuous. The reason it fails to converge uniformly is that for inputs where two coordinates are almost equal (and one is the maximum), the arg max is the index of one or the other, so a small change in input yields a large change in output. For example,${\displaystyle {\sigma }_{\beta }(1,1.0001)\to (0,1),}$ but${\displaystyle {\sigma }_{\beta }(1,0.9999)\to (1,0),}$ and${\displaystyle {\sigma }_{\beta }(1,1)=1/2}$ for all inputs: the closer the points are to the singular set${\displaystyle (x,x)}$, the slower they converge. However, softargmax doesconverge compactly on the non-singular set.

Conversely, as⁠${\displaystyle \beta \to -\mathrm{\infty }}$⁠, softargmax converges to arg min in the same way, where here the singular set is points with two argmin values. In the language oftropical analysis, the softmax is adeformation or "quantization" of arg max and arg min, corresponding to using thelog semiring instead of themax-plus semiring (respectivelymin-plus semiring), and recovering the arg max or arg min by taking the limit is called "tropicalization" or "dequantization".

It is also the case that, for any fixedβ, if one input⁠${\displaystyle z_i}$⁠ is much larger than the othersrelative to the temperature,${\displaystyle T=1/\beta }$, the output is approximately the arg max. For example, a difference of 10 is large relative to a temperature of 1:$${\displaystyle \sigma (0,10):={\sigma }_1(0,10)=\left(1/\left(1+e^{10}\right),e^{10}/\left(1+e^{10}\right)\right)\approx (0.00005,0.99995)}$$However, if the difference is small relative to the temperature, the value is not close to the arg max. For example, a difference of 10 is small relative to a temperature of 100:$${\displaystyle {\sigma }_{1/100}(0,10)=\left(1/\left(1+e^{1/10}\right),e^{1/10}/\left(1+e^{1/10}\right)\right)\approx (0.475,0.525).}$$As⁠${\displaystyle \beta \to \mathrm{\infty }}$⁠, temperature goes to zero,${\displaystyle T=1/\beta \to 0}$, so eventually all differences become large (relative to a shrinking temperature), which gives another interpretation for the limit behavior.

Instatistical mechanics, the softargmax function is known as theBoltzmann distribution (orGibbs distribution):^[5]: 7 the index set${\displaystyle 1,\dots ,k}$ are themicrostates of the system; the inputs${\displaystyle z_i}$ are the energies of that state; the denominator is known as thepartition function, often denoted byZ; and the factorβ is called thecoldness (orthermodynamic beta, orinverse temperature).

The softmax function is used in variousmulticlass classification methods, such asmultinomial logistic regression (also known as softmax regression),^{[2]: 206–209 [6]} multiclasslinear discriminant analysis,naive Bayes classifiers, andartificial neural networks.^[7] Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result ofK distinctlinear functions, and the predicted probability for thejth class given a sample vectorx and a weighting vectorw is:

$${\displaystyle P(y=j\mid \mathbf{x})=\frac{e^{{\mathbf{x}}^{\mathsf{T}}{\mathbf{w}}_j}}{\sum _{k=1}^K{e}^{{\mathbf{x}}^{\mathsf{T}}{\mathbf{w}}_k}}}$$

This can be seen as thecomposition ofK linear functions${\displaystyle \mathbf{x}\mapsto {\mathbf{x}}^{\mathsf{T}}{\mathbf{w}}_1,\dots ,\mathbf{x}\mapsto {\mathbf{x}}^{\mathsf{T}}{\mathbf{w}}_K}$ and the softmax function (where${\displaystyle {\mathbf{x}}^{\mathsf{T}}\mathbf{w}}$ denotes the inner product of${\displaystyle \mathbf{x}}$ and${\displaystyle \mathbf{w}}$). The operation is equivalent to applying a linear operator defined by${\displaystyle \mathbf{w}}$ to vectors${\displaystyle \mathbf{x}}$, thus transforming the original, probably highly-dimensional, input to vectors in aK-dimensional space${\displaystyle {\mathbb{R}}^K}$.

The standard softmax function is often used in the final layer of a neural network-based classifier. Such networks are commonly trained under alog loss (orcross-entropy) regime, giving a non-linear variant of multinomial logistic regression.

Since the function maps a vector and a specific index${\displaystyle i}$ to a real value, the derivative needs to take the index into account:

$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{q}_k}\sigma (\mathbf{\text{q}},i)=\sigma (\mathbf{\text{q}},i)({\delta }_{ik}-\sigma (\mathbf{\text{q}},k)).}$$

This expression is symmetrical in the indexes${\displaystyle i,k}$ and thus may also be expressed as

$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{q}_k}\sigma (\mathbf{\text{q}},i)=\sigma (\mathbf{\text{q}},k)({\delta }_{ik}-\sigma (\mathbf{\text{q}},i)).}$$

Here, theKronecker delta is used for simplicity (cf. the derivative of asigmoid function, being expressed via the function itself).

To ensure stable numerical computations subtracting the maximum value from the input vector is common. This approach, while not altering the output or the derivative theoretically, enhances stability by directly controlling the maximum exponent value computed.

If the function is scaled with the parameter${\displaystyle \beta }$, then these expressions must be multiplied by${\displaystyle \beta }$.

Seemultinomial logit for a probability model which uses the softmax activation function.

In the field ofreinforcement learning, a softmax function can be used to convert values into action probabilities. The function commonly used is:^[8]$${\displaystyle P_t(a)=\frac{\exp (q_t(a)/\tau )}{\sum _{i=1}^n\exp (q_t(i)/\tau )}\text{,}}$$

where the action value${\displaystyle q_t(a)}$ corresponds to the expected reward of following action a and${\displaystyle \tau }$ is called a temperature parameter (in allusion tostatistical mechanics). For high temperatures (${\displaystyle \tau \to \mathrm{\infty }}$), all actions have nearly the same probability and the lower the temperature, the more expected rewards affect the probability. For a low temperature (${\displaystyle \tau \to 0^{+}}$), the probability of the action with the highest expected reward tends to 1.

In neural network applications, the numberK of possible outcomes is often large, e.g. in case ofneural language models that predict the most likely outcome out of a vocabulary which might contain millions of possible words.^[9] This can make the calculations for the softmax layer (i.e. the matrix multiplications to determine the${\displaystyle z_i}$, followed by the application of the softmax function itself) computationally expensive.^[9][10] What's more, thegradient descentbackpropagation method for training such a neural network involves calculating the softmax for every training example, and the number of training examples can also become large. The computational effort for the softmax became a major limiting factor in the development of larger neural language models, motivating various remedies to reduce training times.^[9][10]

Approaches that reorganize the softmax layer for more efficient calculation include thehierarchical softmax and thedifferentiated softmax.^[9] The hierarchical softmax (introduced by Morin andBengio in 2005) uses a binary tree structure where the outcomes (vocabulary words) are the leaves and the intermediate nodes are suitably selected "classes" of outcomes, forminglatent variables.^[10][11] The desired probability (softmax value) of a leaf (outcome) can then be calculated as the product of the probabilities of all nodes on the path from the root to that leaf.^[10] Ideally, when the tree is balanced, this would reduce thecomputational complexity from${\displaystyle O(K)}$ to${\displaystyle O({\log }_{2}K)}$.^[11] In practice, results depend on choosing a good strategy for clustering the outcomes into classes.^[10][11] AHuffman tree was used for this in Google'sword2vec models (introduced in 2013) to achieve scalability.^[9]

A second kind of remedies is based on approximating the softmax (during training) with modified loss functions that avoid the calculation of the full normalization factor.^[9] These include methods that restrict the normalization sum to a sample of outcomes (e.g. Importance Sampling, Target Sampling).^[9][10]

The standard softmax is numerically unstable because of large exponentiations. Thesafe softmax method calculates instead$${\displaystyle \sigma (\mathbf{z}{)}_i=\frac{e^{\beta (z_i-m)}}{\sum _{j=1}^K{e}^{\beta (z_j-m)}}}$$where${\displaystyle m=\underset{i}{max}{z}_i}$ is the largest factor involved. Subtracting by it guarantees that the exponentiations result in at most 1.

Theattention mechanism inTransformers takes three arguments: a "query vector"${\displaystyle q}$, a list of "key vectors"${\displaystyle k_1,\dots ,k_N}$, and a list of "value vectors"${\displaystyle v_1,\dots ,v_N}$, and outputs a softmax-weighted sum over value vectors:$${\displaystyle o=\sum _{i=1}^N\frac{e^{q^T{k}_i-m}}{\sum _{j=1}^N{e}^{q^T{k}_j-m}}{v}_i}$$The standard softmax method involves several loops over the inputs, which would bebottlenecked by memory bandwidth. TheFlashAttention method is acommunication-avoiding algorithm that fuses these operations into a single loop, increasing thearithmetic intensity. It is anonline algorithm that computes the following quantities:^[12][13]and returns${\displaystyle o_N/l_N}$. In practice, FlashAttention operates over multiple queries and keys per loop iteration, in a similar way asblocked matrix multiplication. Ifbackpropagation is needed, then the output vectors and the intermediate arrays${\displaystyle [m_1,\dots ,m_N],[l_1,\dots ,l_N]}$ are cached, and during the backward pass, attention matrices arerematerialized from these, making it a form of gradient checkpointing.

Geometrically the softmax function maps thevector space${\displaystyle {\mathbb{R}}^K}$ to theboundary of thestandard${\displaystyle (K-1)}$-simplex, cutting the dimension by one (the range is a${\displaystyle (K-1)}$-dimensional simplex in${\displaystyle K}$-dimensional space), due to thelinear constraint that all output sum to 1 meaning it lies on ahyperplane.

Along the main diagonal${\displaystyle (x,x,\dots ,x),}$ softmax is just the uniform distribution on outputs,${\displaystyle (1/n,\dots ,1/n)}$: equal scores yield equal probabilities.

More generally, softmax is invariant under translation by the same value in each coordinate: adding${\displaystyle \mathbf{c}=(c,\dots ,c)}$ to the inputs${\displaystyle \mathbf{z}}$ yields${\displaystyle \sigma (\mathbf{z}+\mathbf{c})=\sigma (\mathbf{z})}$, because it multiplies each exponent by the same factor,${\displaystyle e^c}$ (because${\displaystyle e^{z_i+c}=e^{z_i}\cdot e^c}$), so the ratios do not change:$${\displaystyle \sigma (\mathbf{z}+\mathbf{c}{)}_j=\frac{e^{z_j+c}}{\sum _{k=1}^K{e}^{z_k+c}}=\frac{e^{z_j}\cdot e^c}{\sum _{k=1}^K{e}^{z_k}\cdot e^c}=\sigma (\mathbf{z}{)}_j.}$$

Geometrically, softmax is constant along diagonals: this is the dimension that is eliminated, and corresponds to the softmax output being independent of a translation in the input scores (a choice of 0 score). One can normalize input scores by assuming that the sum is zero (subtract the average:${\displaystyle \mathbf{c}}$ where$c=\frac{1}{n}\sum z_i$), and then the softmax takes the hyperplane of points that sum to zero,$\sum z_i=0$, to the open simplex of positive values that sum to 1$\sum \sigma (\mathbf{z}{)}_i=1$, analogously to how the exponent takes 0 to 1,${\displaystyle e^0=1}$ and is positive.

By contrast, softmax is not invariant under scaling. For instance,${\displaystyle \sigma ((0,1))=(1/(1+e),e/(1+e))}$ but${\displaystyle \sigma ((0,2))=(1/\left(1+e^2\right),e^2/\left(1+e^2\right)).}$

Thestandard logistic function is the special case for a 1-dimensional axis in 2-dimensional space, say thex-axis in the(x, y) plane. One variable is fixed at 0 (say${\displaystyle z_2=0}$), so${\displaystyle e^0=1}$, and the other variable can vary, denote it${\displaystyle z_1=x}$, so$e^{z_1}/\sum _{k=1}^2{e}^{z_k}=e^x/\left(e^x+1\right),$ the standard logistic function, and$e^{z_2}/\sum _{k=1}^2{e}^{z_k}=1/\left(e^x+1\right),$ its complement (meaning they add up to 1). The 1-dimensional input could alternatively be expressed as the line${\displaystyle (x/2,-x/2)}$, with outputs${\displaystyle e^{x/2}/\left(e^{x/2}+e^{-x/2}\right)=e^x/\left(e^x+1\right)}$ and${\displaystyle e^{-x/2}/\left(e^{x/2}+e^{-x/2}\right)=1/\left(e^x+1\right).}$

The softmax function is also the gradient of theLogSumExp function:$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{z}_i}\mathrm{LSE}(\mathbf{z})=\frac{\exp z_i}{\sum _{j=1}^K\exp z_j}=\sigma (\mathbf{z}{)}_i,\text{ for }i=1,\dots ,K,\mathbf{z}=(z_1,\dots ,z_K)\in {\mathbb{R}}^K,}$$where the LogSumExp function is defined as${\displaystyle \mathrm{LSE}(z_1,\dots ,z_n)=\log \left(\exp (z_1)+\cdots +\exp (z_n)\right)}$.

The gradient of softmax is thus${\displaystyle {\mathrm{\partial }}_{z_j}{\sigma }_i={\sigma }_i({\delta }_{ij}-{\sigma }_j)}$.

The softmax function was used instatistical mechanics as theBoltzmann distribution in the foundational paperBoltzmann (1868),^[14] formalized and popularized in the influential textbookGibbs (1902).^[15]

The use of the softmax indecision theory is credited toR. Duncan Luce,^[16]: 1 who used the axiom ofindependence of irrelevant alternatives inrational choice theory to deduce the softmax inLuce's choice axiom for relative preferences.^{[citation needed]}

In machine learning, the term "softmax" is credited to John S. Bridle in two 1989 conference papers,Bridle (1990a):^[16]: 1 andBridle (1990b):^[3]

We are concerned with feed-forward non-linear networks (multi-layer perceptrons, or MLPs) with multiple outputs. We wish to treat the outputs of the network as probabilities of alternatives (e.g. pattern classes), conditioned on the inputs. We look for appropriate output non-linearities and for appropriate criteria for adaptation of the parameters of the network (e.g. weights). We explain two modifications: probability scoring, which is an alternative to squared error minimisation, and a normalised exponential (softmax) multi-input generalisation of the logistic non-linearity.^[17]: 227

For any input, the outputs must all be positive and they must sum to unity. ...

Given a set of unconstrained values,⁠${\displaystyle V_j(x)}$⁠, we can ensure both conditions by using a Normalised Exponential transformation:$${\displaystyle Q_j(x)=e^{V_j(x)}/\sum _k{e}^{V_k(x)}}$$This transformation can be considered a multi-input generalisation of the logistic, operating on the whole output layer. It preserves the rank order of its input values, and is a differentiable generalisation of the 'winner-take-all' operation of picking the maximum value. For this reason we like to refer to it assoftmax.^[18]: 213

With an input of(1, 2, 3, 4, 1, 2, 3), the softmax is approximately(0.024, 0.064, 0.175, 0.475, 0.024, 0.064, 0.175). The output has most of its weight where the "4" was in the original input. This is what the function is normally used for: to highlight the largest values and suppress values which are significantly below the maximum value. But note: a change oftemperature changes the output. When the temperature is multiplied by 10, the inputs are effectively(0.1, 0.2, 0.3, 0.4, 0.1, 0.2, 0.3) and the softmax is approximately(0.125, 0.138, 0.153, 0.169, 0.125, 0.138, 0.153). This shows that high temperatures de-emphasize the maximum value.

Computation of this example usingPython code:

>>>importnumpyasnp>>>z=np.array([1.0,2.0,3.0,4.0,1.0,2.0,3.0])>>>beta=1.0>>>np.exp(beta*z)/np.sum(np.exp(beta*z))array([0.02364054, 0.06426166, 0.1746813, 0.474833, 0.02364054,       0.06426166, 0.1746813])

The softmax function generates probability predictions densely distributed over itssupport. Other functions likesparsemax or α-entmax can be used when sparse probability predictions are desired.^[19] Also theGumbel-softmax reparametrization trick can be used when sampling from a discrete-discrete distribution needs to be mimicked in a differentiable manner.

• Softplus
• Multinomial logistic regression
• Dirichlet distribution – an alternative way to sample categorical distributions
• Partition function
• Exponential tilting – a generalization of Softmax to more general probability distributions

• Computational neuroscience
• Logistic regression
• Artificial neural networks
• Functions and mappings
• Exponentials

• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from March 2024
• Articles with example Python (programming language) code
• Articles with example Julia code
• Articles with example R code