Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

331–340 of 347 posts

Re: Attention Is Off By One

#331

Post by jszymborski »

thesausageking wrote: post #320
tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

This is a technique that's been known for years and is in PyTorch. It's not widely used because people tried it and, in practice, it doesn't work as well. OP calling it a "bug that's been overlooked for 8+ years" is click bait.

> ...is in PyTorch

Could anyone kindly point me to this as I can't find it.

Re: Attention Is Off By One

#332

Post by TikiTDO »

bambax wrote: post #266
jablongo wrote: post #239

Earlier quoted context omitted.

Yea - the way we can "tell if this is good" is by a) train two identical models on a large dataset, one with the +1 in the denominator for the softmax steps of the attention modules, one without b) show that they have similar performance (doubt the +1 will make performance better, but we need to show it doesn't make things worse) c) show that there are less "blowups" in the model with +1, and therefore they are more…

> train two identical models on a large dataset Yes but how much would this cost? Would it be possible to build a small dataset that produces known outlier values, and test on that?

It doesn't need to be two huge models. If there is an advantage to doing this, I'd expect that you would see it even in a small test case. I'm sure we'll see something by the end of the week if not earlier if there's something to it.

Re: Attention Is Off By One

#333

Post by coldtea »

dragonwriter wrote: post #322
coldtea wrote: post #319

Earlier quoted context omitted.

> Leaving aside a debate over whether her work is a good match for “woke”, she very much cannot have been woke before woke was a thing A game of being pedantic is always welcome: She very well could have been "woke before woke was a thing", because "woke" as the parent means it in her case, refers to the modern usage (of like, 2 decades), not the original term of the 40s that might have preceeded her birth. So take t…

The 1938 sense in which it was coin3d is exactly the sense of the 1950s and the sense that got ibcreased attention circa 2000s and catapulted to attention alongside BLM (which itself was a response to the same kind of even that the art in which the phrase was coined for responded to). The only newer sense is the American Right’s use of the term to replace “political correctness” as an empty epithet for everything and…

Language grows organically, and the American right gets as much as the American left to define what a word means or how proponents of a movement or social fad are seen in practice (besides woke's standard definition is just "awake", if someone insists on the "original meaning")

So, one side could see woke in theory as a noble activist/social consciousness practice, which can not go wrong and helps liberate us all.

The other side might see woke in practice as intolerable virtue signalling and self-aggrandizing whose actions often border on farcical.

Re: Attention Is Off By One

#334

Post by thomasahle »

dijksterhuis wrote: post #314
thomasahle wrote: post #224

Earlier quoted context omitted.

Are dummy tokens just tokens that don't have an associated input/output token? Like, a way to give more computational power to the model without splitting the text into more actual tokens?

TL;DR sort of yes. But they're also useful for reasons not related to computational "power". An example here with an actual algorithm, although it's been a couple of years so my explanation might be a bit wrong in places. and/or i might have gotten the completely wrong end of the stick with the current thread. -- The CTC (Connectionist Temporal Classification [0]) algorithm maps a sequence x with length X -> sequence…

> By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) > y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d

This is a great solution. Though that's a dummy token in the output rather than the input. I guess you could do something inverse to do text to speech, but it might be hard to say where to insert the dummy tokens in that case.

Re: Attention Is Off By One

#335

Post by jdougan »

RationPhantoms wrote: post #216
smallnamespace wrote: post #159

Earlier quoted context omitted.

Papers are mostly read by other researchers, where the added background is actively bad because it obscures the real meat of the paper to the main audience. If you just wanted a digestible intro then you would usually buy a textbook. I think the argument that every research paper ought to be a mashup of a textbook + the actual research to be a bit silly from a “people should specialize at what they’re good at” standp…

I've long wanted an informational slider to bits of text. Something where you can zoom in and out to the level of desired complexity. LLM's might be able to fill in some of those gaps. You could turn any paper into a introduction of the subject it's a part of.

Look for "stretchtext"

Re: Attention Is Off By One

#336

Post by jablongo »

TikiTDO wrote: post #332
bambax wrote: post #266

Earlier quoted context omitted.

> train two identical models on a large dataset Yes but how much would this cost? Would it be possible to build a small dataset that produces known outlier values, and test on that?

It doesn't need to be two huge models. If there is an advantage to doing this, I'd expect that you would see it even in a small test case. I'm sure we'll see something by the end of the week if not earlier if there's something to it.

The reason it might need to be huge is because the long tail of extreme weights might only begin to show up then, but yes best to just start w something you can run on a laptop.

Re: Attention Is Off By One

#337

Post by numeri »

TikiTDO wrote: post #332
bambax wrote: post #266

Earlier quoted context omitted.

> train two identical models on a large dataset Yes but how much would this cost? Would it be possible to build a small dataset that produces known outlier values, and test on that?

It doesn't need to be two huge models. If there is an advantage to doing this, I'd expect that you would see it even in a small test case. I'm sure we'll see something by the end of the week if not earlier if there's something to it.

One of the most significant quantization papers of the last year [1] found precisely that these outliers only start occuring with LLMs at 6.7B parameters and above.

One of the most important keys to the success of deep learning in the last couple years has been the fact that emergent features exist after certain scales, so I wouldn't be too quick to dismiss things that don't help at smaller scales, nor would I be certain that all the tricks that help in small data/parameter regimes will necessarily help in larger models. Unfortunately!

[1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea...

Re: Attention Is Off By One

#338

Post by TikiTDO »

numeri wrote: post #337
TikiTDO wrote: post #332

Earlier quoted context omitted.

It doesn't need to be two huge models. If there is an advantage to doing this, I'd expect that you would see it even in a small test case. I'm sure we'll see something by the end of the week if not earlier if there's something to it.

One of the most significant quantization papers of the last year [1] found precisely that these outliers only start occuring with LLMs at 6.7B parameters and above. One of the most important keys to the success of deep learning in the last couple years has been the fact that emergent features exist after certain scales, so I wouldn't be too quick to dismiss things that don't help at smaller scales, nor would I be cer…

Looking at that paper, they appear to be saying that 6.7B is where the problem becomes so intense that no single quantization method can keep up. From what I gather, the paper claims that such outliers start occur down to 125M param models, then at around 1.3B they begin to affect the FFN, and at around 6.7B is when the issue really starts to become apparent because "100% of layers use the same dimension for outliers."

So while you obviously wouldn't be able to conclusively prove the idea fixes the issue in larger models, if you know what you are looking for you should be able to validate that the method works in general down to very small models.

That said, consumer grade cards should be able to train an 8B model with quantization, so you might as well train the whole thing.

Re: Attention Is Off By One

#339

Post by sriku »

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

Don't most softmax implementations include an epsilon in the denominator which likely serves the same purpose? So the suggestion is to set that epsilon to 1?

Re: Attention Is Off By One

#340

Post by GTP »

thfuran wrote: post #308
GTP wrote: post #296

Earlier quoted context omitted.

For the spaces and for some (maybe most?) languages you don't even need a NN to add spaces: as words made of two or more words aren't that common, and when those occur you probably want to use the composite one, it boils down to start from the beginning of the text and look in a dictionary what's the longest string that is a valid word. The only language that I know of that uses a lot of composite words (I mean words…

I think you're significantly underestimating how many words could be retokenized into multiple words even before considering how concatenation affects things. For example: Concatenate is a word, but so are con, catenate, cat, and enate. Yes, no two of those are likely to be used in sequence, but I don't think that's a very reliable rule overall—"a" and "an" are both common words and negative prefixes.

Maybe you're right. I was biased by my native language, which doesn't have the a/an problem that English has.
Post reply on HN