What happens if you use an integer like 2 or 3 instead of e in the softmax equation? Is e what makes it so they end up summing to 1? (I have not done real math in yearssss.)
Softmax, can you derive the Jacobian? And should you care?
31–40 of 50 posts
Re: Softmax, can you derive the Jacobian? And should you care?
#32One thing extremely worth noting that the article does not: The reason "temperature" is called such is because softmax is mathematically identical to the Boltzmann distribution [1] from thermodynamics, which describes the probability distribution of energy states of an ensemble of particles in equilibrium. In terminology more well understood by ML folks, the particles' energies will be distributed as the softmax of t…
The article also does not discuss the awesomeness of LLMs at negative temperatures!
Re: Softmax, can you derive the Jacobian? And should you care?
#33Something that really helped me grasp the foundational relevance of the softmax is to justify from first principles why e^x shows up in the preferred mapping function in the numerator (1). The stated problem of mapping raw inputs/scores/logits to a probability distribution can be solved by a bunch of arbitrary functions, and the usual justification given for a softmax is "it has nice derivatives" which is empirically…
> The stated problem of mapping raw inputs/scores/logits to a probability distribution can be solved by a bunch of arbitrary functions, and the usual justification given for a softmax is "it has nice derivatives" which is empirically useful but not satisfying. Often there isn't any more to it than that. For example, the entire justification for least-squares error measurement is that it has convenient derivatives.
The central limit theorem is an extremely powerful justification. That doesn't mean it's considered whenever it's used, but it absolutely can be strongly justified (to the degree that other error measurements are only needed in relatively small samples of the feature space where errors will not yet converge to Gaussian)
Re: Softmax, can you derive the Jacobian? And should you care?
#34Earlier quoted context omitted.
The softmax, after the network has been trained, yields an estimate of the probability in the training data, but it is not that probability itself.
Which models are not trained with the log softmax as the loss function?
Softmax isn't a loss function. It is used to transform model outputs into positive numbers that sum to 1, so that they can be interpreted as probabilities, and then those numbers are passed into (typically) the cross entropy loss function. I think you mean, which models are trained using some function other than softmax to transform the model outputs. There are a number of alternatives to softmax, such as the ones described here https://www.emergentmind.com/topics/sparsemax
Re: Softmax, can you derive the Jacobian? And should you care?
#35What happens if you use an integer like 2 or 3 instead of e in the softmax equation? Is e what makes it so they end up summing to 1? (I have not done real math in yearssss.)
That's equivalent to changing the temperature.
Also, so long as the function is non-negative for all inputs and positive for at least one you'll always get a valid probability distribution.
Re: Softmax, can you derive the Jacobian? And should you care?
#36Earlier quoted context omitted.
Which models are not trained with the log softmax as the loss function?
Softmax isn't a loss function. It is used to transform model outputs into positive numbers that sum to 1, so that they can be interpreted as probabilities, and then those numbers are passed into (typically) the cross entropy loss function. I think you mean, which models are trained using some function other than softmax to transform the model outputs. There are a number of alternatives to softmax, such as the ones de…
The cross entropy loss function is softmax. They are one and the same.
Re: Softmax, can you derive the Jacobian? And should you care?
#37What happens if you use an integer like 2 or 3 instead of e in the softmax equation? Is e what makes it so they end up summing to 1? (I have not done real math in yearssss.)
It's equivalent to multiplying all inputs by log b. And multiplying all inputs by a value changes how much the probabilities are extremized. This is easy to see because adding a value to everything doesn't change the output, so the biggest input can be assumed to be 0 and others negative. So multiplying by 0 makes all outputs equal while as the multiplier tends to infinity, all other inputs tend to -infinity and thus the biggest output tends to 1 and others to 0. Multiplying by negative numbers results in the lowest becoming the highest.
Re: Softmax, can you derive the Jacobian? And should you care?
#38What happens if you use an integer like 2 or 3 instead of e in the softmax equation? Is e what makes it so they end up summing to 1? (I have not done real math in yearssss.)
It works the same way:
softmax is essentially just applying the normalization to the vector exp(x).
From an "engineering" POV this effectively ensures that the vector you normalize has strictly positive entries, so the result ends up being a proper distribution.
From a theory POV you get softmax like distributions (Gibbs distributions) by trying to balance following some energy E(x) and the entropy of the distribution. In essence the softmax is the answer to "I try to follow the maximum of a function E(x) but I need to maintain some level of uncertainy".
The balancing coefficient between entropy and picking the maximum of the function is called "temperature" (following the behavior of particles in a physical system: The colder the system, the lower the chance of having particles randomly walk away from the minimal energy state).
specifically, the temperature is
softmax(x/temp)
if you draw temp->0, your softmax slowly becomes an argmax (with temp=0 being a literal argmax). If you increase the temperature, you are closer to the "random fluctuations" leaving more room for sampling x values that are not the maximum of x. (this is why e.g. LLMs become deterministic as you decrease temp->0)
Using a different base other than e implicitly changes the temperature:
N^x = exp(ln(N) x)
The normalization works the same since you are still dividing a positive value N^x by the sum of all alternatives sum(N^x_i), which is a normalization by design
Re: Softmax, can you derive the Jacobian? And should you care?
#39Earlier quoted context omitted.
Softmax isn't a loss function. It is used to transform model outputs into positive numbers that sum to 1, so that they can be interpreted as probabilities, and then those numbers are passed into (typically) the cross entropy loss function. I think you mean, which models are trained using some function other than softmax to transform the model outputs. There are a number of alternatives to softmax, such as the ones de…
The cross entropy loss function is softmax . They are one and the same.
They’re not. Cross entropy loss is E[-log q] where q is a probability. You could convert the model outputs x into probabilities using some other function like q = 1/Z x^2, and compute cross entropy loss just fine.
Re: Softmax, can you derive the Jacobian? And should you care?
#40Earlier quoted context omitted.
The cross entropy loss function is softmax . They are one and the same.
They’re not. Cross entropy loss is E[-log q] where q is a probability. You could convert the model outputs x into probabilities using some other function like q = 1/Z x^2, and compute cross entropy loss just fine.
Behold the softmax: https://docs.pytorch.org/docs/2.11/generated/torch.nn.CrossE...