Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

71–80 of 347 posts

Re: Attention Is Off By One

#71

Post by joebiden2 »

Legend2440 wrote: post #46
fwlr wrote: post #24

The author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enoug…

Yeah, but it lacks the most important test: results. He hasn't actually tried it, he just thinks it will work. For such a simple change to the softmax it wouldn't take long to verify. It's really embarrassing to not do that before publishing.

You seem to really disregard the positions of this author. They seem to have invested substantial efforts in that specific area of research.

To validate the idea the author has, it would be required to train a LLM from zero. If the author is right, you would get similar results to the current generation of LLMs, but with (a lot) less space required for the intermediate layers.

The time to achieve that is still measured in kilo- to mega-dollars, why is it wrong to put that idea in the open to substantially criticize or adopt?

Re: Attention Is Off By One

#72

Post by Legend2440 »

joebiden2 wrote: post #71
Legend2440 wrote: post #46

Earlier quoted context omitted.

Yeah, but it lacks the most important test: results. He hasn't actually tried it, he just thinks it will work. For such a simple change to the softmax it wouldn't take long to verify. It's really embarrassing to not do that before publishing.

You seem to really disregard the positions of this author. They seem to have invested substantial efforts in that specific area of research. To validate the idea the author has, it would be required to train a LLM from zero. If the author is right, you would get similar results to the current generation of LLMs, but with (a lot) less space required for the intermediate layers. The time to achieve that is still measur…

You don't need to train a ChatGPT-sized LLM, a toy nanoGPT would have been enough. You can train those on a consumer GPU in an afternoon.

And yes I do disregard his research effort. There are hundreds of well-justified and well-researched "clever tricks" for improving Transformers, and almost all of them don't work. I'll believe it when I see the results.

Re: Attention Is Off By One

#73

Post by simbolit »

gremlinsinc wrote: post #11
simbolit wrote: post #5

I don't understand this well enough to say if it is correct, but I do understand it well enough to say it is important if correct.

I don't know half of you half as well as I should like; and I like less than half of you half as well as you deserve

I don't understand your comment, but I gather that you and others didn't like mine, which is noted, I will try to do better.

Re: Attention Is Off By One

#74

Post by refulgentis »

sebzim4500 wrote: post #70
refulgentis wrote: post #64

Earlier quoted context omitted.

not a token, and not the transformers, but yes, commercial chat models are fine-tuned on text transcripts containing dialogues. (i believe llama-2 was as well)

Are you sure? I have never seen an LLM that did not have a special token for start of text, I'm certain that llama had one and I don't remember anywhere in the llama-2 paper where they said they removed it.

tl;dr: you're right

it's messy though, bear with me for the full explanation:

- your initial post says "" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI

- there is a bo_S_ token, which acts as you described

- I averaged my attention over your post and the initial reply, which answers as if you were using "" in the misunderstood way

- when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand as a way to denote special tokens, not necessarily an XML tag

Re: Attention Is Off By One

#75

Post by Legend2440 »

ambrozk wrote: post #59
Legend2440 wrote: post #40

Earlier quoted context omitted.

Counterargument: this blogpost is worthless. You get all the way to the end and then find out he hasn't actually tried it, not even on a toy model. It's just a neat idea he thinks will work.

Why would that make it worthless?

Because until he tries it, who knows if it works?

There are a thousand papers out there making minor tweaks to the transformer architecture. 99% of them are also worthless and forgotten.

Re: Attention Is Off By One

#76

Post by Vervious »

r3trohack3r wrote: post #55
doliveira wrote: post #30

Earlier quoted context omitted.

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

There was this sociologist who had written a paper for us all to read ahead of time. I started to read the damn thing, and my eyes were coming out: I couldn’t make head nor tail of it! I figured it was because I hadn’t read any of the books on the list. I had this uneasy feeling of “I’m not adequate,” until finally I said to myself “I’m gonna stop, and read one sentence slowly so I can figure out what the hell it mea…

Systems research papers do not represent all research papers out there, not even in computer science.

In cryptography, certainly a paper with formal definitions and proofs can be much more valuable than a corresponding blog post. It's a field where formalism is desired, if not necessary. Otherwise you can't check other people's "proofs", or even know what model you're working in.

I think, since people haven't come up with better formalisms, sometimes it's quite obtuse, which gets mistaken as "academic writing", when really it's a best effort to formalize.

Re: Attention Is Off By One

#77

Post by Waterluvian »

mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

To finish the author’s analogy:

Blog posts are written by those who arrive first.

In a weird way my mental model is: blog posts are the recon team discovering a new idea. They might have errors. They might be incomplete. Maybe they’re outright wrong. Stakes are lower as it took less effort to get there and less loss if a position is abandoned.

Then papers are authored, often much later, and they’re the regulars coming in to fortify a newly captured idea. They provide (or at least are supposed to) rigor to the idea. A fortification of a position that we decide is worth holding.

Yeah, this analogy is probably sloppy. But in my brain there’s an eternal conflict against ignorance as we keep advancing into the unknown.

Re: Attention Is Off By One

#78

Post by janalsncm »

WithinReason wrote: post #56
janalsncm wrote: post #27

Earlier quoted context omitted.

That isn’t true at all. Train a smaller model on a smaller dataset. You can even train on your laptop. It’s definitely feasible. This is just a proof of concept, it doesn’t need to beat state of the art.

Maybe I edited my comment too late.

> I believe the outlier problem that this solves only appears for very large models.

Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.

Re: Attention Is Off By One

#79

Post by PoignardAzur »

ambrozk wrote: post #59
Legend2440 wrote: post #40

Earlier quoted context omitted.

Counterargument: this blogpost is worthless. You get all the way to the end and then find out he hasn't actually tried it, not even on a toy model. It's just a neat idea he thinks will work.

Why would that make it worthless?

Among other reasons, because the decoder-only version of the original transformer architecture has proven weirdly resistant to these kinds of hacks and clever optimizations.

Ideas like sparse attention, tree attention, residual attention, etc, all sound good on paper, but when researchers try to reproduce them they either find no results or results that don't scale. Even AliBi is turning out to be less powerful than scaled-down positional embeddings. It's almost a bitter lesson on its own: you can't beat the original transformer.

Optimizations that do stick around tend to be the ones that preserve the original algorithm but help with caching or memory accesses.

Re: Attention Is Off By One

#80

Post by Imnimo »

marcyb5st wrote: post #66
Imnimo wrote: post #50

>The problem with using softmax is that it forces each attention head to make an annotation, even if it has no information to add to the output vector. Using softmax to choose among discrete alternatives is great; using it for optional annotation (i.e. as input into addition) is, like, not cool, man. The problem here is exacerbated with multi-head attention, as a specialized head is more likely to want to “pass” than…

But you are wasting some of the model's capacity to learn to ignore some of that information. I think it wouldn't hurt. However, if I followed the reasoning correctly, I think the biggest win is to reduce the range of the weights more than improving performance. > This is what’s been happening in LLMs – for reasons that are only partially understood, Transformer models contain these outlier weights and are emitting B…

Right, I get the goal of removing the outlier activations, but I just don't understand why outlier activations are a consequence of the model trying to "pass". The story from the linked paper earlier in the post (https://arxiv.org/pdf/2306.12929.pdf) is that the model is doing the following:

-Learn a near-zero representation for some otherwise low-importance token, like delimiters or whitespace.

-When a head wants to "pass", emit an outlier activation to attend to that token nearly-exclusively.

But I'm surprised the model can't just use its existing tools (the post-concat projection layer and the following MLP block) to achieve the same thing. And if the answer is that it could do that, but tends to learn to use the outlier activation trick instead, will giving it a new tool that still allows the use of outlier activations be sufficient?

Post reply on HN