Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

181–190 of 347 posts

Re: Attention Is Off By One

#181

Post by moffkalast »

nl wrote: post #174
quickthrower2 wrote: post #173

I wonder why the author didn't try it on some metal? What is the minimum cost to get information that would say "this is better" or at least "this has a good chance of being better, spending a million training a bigger model is worth it"? Could you rent a bank for 8 A100's for a day and try it out on a smaller model and prove something. Not cheap, but doesn't need VC money either. Probably about $400 on LambdaLabs to…

You could test this on a toy Transformer-based model trainable on a consumer GPU.

It would be hard to say if either of the two completely crap models is more or less crap though. Maybe by repeating it and seeing consistent results despite changing other variables I guess?

Re: Attention Is Off By One

#182

Post by quickthrower2 »

ambrozk wrote: post #59
Legend2440 wrote: post #40

Earlier quoted context omitted.

Counterargument: this blogpost is worthless. You get all the way to the end and then find out he hasn't actually tried it, not even on a toy model. It's just a neat idea he thinks will work.

Why would that make it worthless?

With say system architecture, you can muse on stuff like "well if Kubernetes made this decision, it would definitely be more secure" or "it would scale up quicker" without empirical evidence and other people could argue "yes I agree because" or "no I don't because"... etc.

With large ML models, there probably is no intuition like this. We just don't know "if I do the common sense thing X, it surely will produce better results for a given benchmark" ... well we have no idea until it is tried out.

Re: Attention Is Off By One

#183

Post by cubefox »

karaterobot wrote: post #154
doliveira wrote: post #30

Earlier quoted context omitted.

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

The writing quality of academic papers is very poor, whatever its intended characteristics are, and we deserve better. I'm skeptical that the only way for them to be precise and technical is to make them impenetrable. I think there is a culture of academic writing (many different cultures, really) that has adopted a voice and writing style which became a parody of itself over time. Here's a trivial example: You frequ…

In philosophy papers you see authors often use the pronoun "I", similar to blog posts. But they have other ways to make them hard to parse for outsiders.

Re: Attention Is Off By One

#184

Post by refulgentis »

Q6T46nT668w6i3m wrote: post #169

This method was frequently used prior to the ubiquity of dummy tokens. XLNet was the paper that introduced me to this idea. I believe it’s been in PyTorch since 2019/2020. I would not be surprised if someone finds an earlier reference. I’m surprised by the pompousness in the OP. Especially about something that most people who do transformer research understand. I’m also surprised that so many in the replies are takin…

He's not being pompous, people appreciate the informality and straightforwardness and self-deprecation, which are the opposite of pompus.

You are reading some of the more ambiguous self-deprecation as genuine claims.

TL;DR on why this is important and he's sharing: it's a sort of niche thing that really only matters if you're trying to run pale imitations of ChatGPT on constrained hardware. That's why it's entirely possible the big guns didn't see it as important, they're not trying to run LLMs on a 3090

Re: Attention Is Off By One

#185

Post by joemckenna »

Whereas vanilla scaled dot-product attention forces outputs to fall in the convex hull of V, your proposal allows them to drift into the convex hull of V+{0}. It's pretty unlikely that the origin is already contained in the convex hull of V especially for long inputs, so your proposal genuinely changes the quality of outputs.

Re: Attention Is Off By One

#186

Post by nl »

renewiltord wrote: post #146
Legend2440 wrote: post #72

Earlier quoted context omitted.

You don't need to train a ChatGPT-sized LLM, a toy nanoGPT would have been enough. You can train those on a consumer GPU in an afternoon. And yes I do disregard his research effort. There are hundreds of well-justified and well-researched "clever tricks" for improving Transformers, and almost all of them don't work. I'll believe it when I see the results.

Do you know of handy testing steps? I suppose I could ask ChatGPT, but if someone has a validated "here, this is how you do it" I have a 3090 that I can do it on, but I'm not keen to debug anything here.

Testing steps (based on thinking about this for 30 seconds - so probably can be improved):

Train a Transformer based model with and without the modified Softmax (Suggestions: GPT-2 or nanoGPT)

Measure performance - I'd probably start with Perplexity and see if there is any difference (we'd expect little difference).

Quantize both models with different quantization strategies.

Measure the perplexity of the quantized models of different sizes. We'd expect the performance to drop off quicker for the non-modified model than the modified one if this is working.

Re: Attention Is Off By One

#187

Post by lmm »

baq wrote: post #120
doliveira wrote: post #30

Earlier quoted context omitted.

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

Leslie Lamport definitely doesn’t share your opinion. A known fact about the Paxos paper is that there are no dumbed down summaries worth reading because the proper thing is so approachable. Not sure if you only have to sound smart if you’ve got nothing to say but certainly feels like it could be the case.

> A known fact about the Paxos paper is that there are no dumbed down summaries worth reading because the proper thing is so approachable.

A known fact is that it's impossible to actually implement it correctly, and the "approachable" paper seems to be a significant factor in this.

Re: Attention Is Off By One

#188

Post by nl »

moffkalast wrote: post #181
nl wrote: post #174

Earlier quoted context omitted.

You could test this on a toy Transformer-based model trainable on a consumer GPU.

It would be hard to say if either of the two completely crap models is more or less crap though. Maybe by repeating it and seeing consistent results despite changing other variables I guess?

Not at all.

I suggest ways to measure it here: https://news.ycombinator.com/item?id=36855881 but the TL;DR is to choose a metric and compare the reduction in performance for quantized versions of the LM compared to the same LM without the modified Softmax.

Re: Attention Is Off By One

#189

Post by thomasahle »

quickthrower2 wrote: post #168
mlyle wrote: post #141

Earlier quoted context omitted.

Yes.

Can you elaborate? (It wouldn't be the first time there was an extraneous feature that no one has every used in some code!)

If you take the inner product between a lot of more or less random vectors (the key and query vectors in attention) most values are going to be close to 0. This means they contribute by e^0 to the denominator. Now, if you have a context length of say 2000, your denominator is already ~ 2000. Increasing it to 2001 doesn't really make a difference.

Adding 1 to the denominator can be useful if you have softmax with just a few options. Not in self-attention where you have thousands.

Re: Attention Is Off By One

#190

Post by a_bonobo »

jshen wrote: post #172
doliveira wrote: post #30

Earlier quoted context omitted.

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

I've read a lot of scientific papers in the comp sci / machine learning space and they are rarely precise. It's been over a decade since I've ready many papers so maybe this has changed, but I remember reading a paper out of Microsoft about how to make spell correcting auto-completion for search, and it was nearly impossible to figure out precisely how it was implemented. Precision would have been achieved easily by…

Ah yes, my old supervisor was very fond of that strategy.

"Make it sound like we do cool stuff; but don't make it so precise that they can re-implement what we do. Let them come to us so we can co-author papers."

Post reply on HN