Viewing profile — apstroll

Extremely doubtful that it boils down to quadratic scaling of attention. That whole issue is a leftover from the days of small bert models with very few parameters. For large model…

Under a crossentropy loss the output activations do absolutely represent a probability distribution, since that is what we're modeling.

The output distribution is deterministic, the output token is sampled from the output distribution, and is therefore not deterministic. Temperature modulates the output distributio…

This paper is doing exactly that though, handwaving with a couple of floats. The paper is just a collection of observations about what their implementation of shapley value analysi…

Cosine Similarity is very much about similarity, but it's quite fickle and indirect. Given a function f(l, r) that measures, say, the logprobability of observing both l and r, and …