Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

171–180 of 347 posts

Re: Attention Is Off By One

#171

Post by joaogui1 »

baq wrote: post #120
doliveira wrote: post #30

Earlier quoted context omitted.

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

Leslie Lamport definitely doesn’t share your opinion. A known fact about the Paxos paper is that there are no dumbed down summaries worth reading because the proper thing is so approachable. Not sure if you only have to sound smart if you’ve got nothing to say but certainly feels like it could be the case.

Paxos is so mistifyingly hard that Raft was invented as part of a project to understand Paxos (and the advisor and proponent of the project was John Ousterhout, who's pretty badass). There are also I believe a few papers trying to trying to explain Paxos more clearly

Re: Attention Is Off By One

#172

Post by jshen »

doliveira wrote: post #30
mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

I've read a lot of scientific papers in the comp sci / machine learning space and they are rarely precise. It's been over a decade since I've ready many papers so maybe this has changed, but I remember reading a paper out of Microsoft about how to make spell correcting auto-completion for search, and it was nearly impossible to figure out precisely how it was implemented. Precision would have been achieved easily by providing code and a sample data set. instead of was a mix of prose and math equations with many gaps where you had to guess how to fill.

Re: Attention Is Off By One

#173

Post by quickthrower2 »

I wonder why the author didn't try it on some metal?

What is the minimum cost to get information that would say "this is better" or at least "this has a good chance of being better, spending a million training a bigger model is worth it"?

Could you rent a bank for 8 A100's for a day and try it out on a smaller model and prove something. Not cheap, but doesn't need VC money either. Probably about $400 on LambdaLabs to test with/without "quiet attention"

The OP can be as sure as he wants, but it is not worth sounding all "I told you so" without a single benchmark. Should be more "what about this - did you miss it ML people...?" in tone.

Re: Attention Is Off By One

#174

Post by nl »

quickthrower2 wrote: post #173

I wonder why the author didn't try it on some metal? What is the minimum cost to get information that would say "this is better" or at least "this has a good chance of being better, spending a million training a bigger model is worth it"? Could you rent a bank for 8 A100's for a day and try it out on a smaller model and prove something. Not cheap, but doesn't need VC money either. Probably about $400 on LambdaLabs to…

You could test this on a toy Transformer-based model trainable on a consumer GPU.

Re: Attention Is Off By One

#175

Post by light_hue_1 »

mlsu wrote: post #15
chessgecko wrote: post #10

Earlier quoted context omitted.

I think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.

Yes, definitely. If he tried to have it published, the lack of experimental results would definitely be a glaring error. But this is still scientific communication. It's really nice that it's legible! > Even though softmax1 is facially quite boring, I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research. If you want to run some experiments and pro…

There's a perfectly good venue for this communication: a workshop.

Workshop submissions often don't need evidence. They just need a small kernel to spur discussion.

Without experiments, there is no hope of publishing this in anything more than a workshop. Nor should there be.

Re: Attention Is Off By One

#176

Post by PaulHoule »

doliveira wrote: post #30
mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

More fundamentally he's postulating that this will work in a blog post but he doesn't do any experiment to prove that it does.

Re: Attention Is Off By One

#177

Post by joaogui1 »

joaogui1 wrote: post #171
baq wrote: post #120

Earlier quoted context omitted.

Leslie Lamport definitely doesn’t share your opinion. A known fact about the Paxos paper is that there are no dumbed down summaries worth reading because the proper thing is so approachable. Not sure if you only have to sound smart if you’ve got nothing to say but certainly feels like it could be the case.

Paxos is so mistifyingly hard that Raft was invented as part of a project to understand Paxos (and the advisor and proponent of the project was John Ousterhout, who's pretty badass). There are also I believe a few papers trying to trying to explain Paxos more clearly

Just as a quick source to my claims:

1. The raft paper is titled "In Search of an Understandable Consensus Algorithm"

2. The abstract of this tutorial on Understanding Paxos https://www.ux.uis.no/~meling/papers/2013-paxostutorial-opod...

3. Lamport's own "Paxos made simple" https://lamport.azurewebsites.net/pubs/paxos-simple.pdf

Re: Attention Is Off By One

#178

Post by herval »

mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

I learned more from this post than a thousand papers. Amazing writing!

Re: Attention Is Off By One

#179

Post by refulgentis »

sidfthec wrote: post #130
com2kid wrote: post #126

Earlier quoted context omitted.

I've told this story before! It used to be writing the outputs from the C/C++ preprocessor (.i files) to disk took forever (5+ minutes IIRC) with Microsoft's compilers. I asked one of the lead compiler developers why, and he waved me away saying it was just really complicated. Around that time a bunch of tools existed for GCC that worked with .i files, but none existed in the Microsoft ecosystem likely because writin…

But that's not the type of story that's being claimed from the person I responded to. Of course the lead developer waved you off. You wondered why things took forever, and the lead developer knew it was a complicated system and figured it wasn't worth their time investigating. It happened to be incorrect, but the lead developer wasn't in denial. They just filtered the issue out because they can't afford to go down ev…

You are ascribing an absurdly maximalist viewpoint to me, one that would be obviously wrong at its face.

I know it's not so confusing as to get that sort of interpretation, because of the score on the comment, and comments like the above that explain to you how this happens.

As a result, I don't feel comfortable providing more detail publicly about my situation. That far off the mark tends to indicate an aggressive rather than curious interlocutor.

I am comfortable building on their example. The particulars of the issue are quite similar in a very helpful way.

I did the investigation, did a fix, worked it up to my manager and my managers manager. Elated, we work diligently for a couple weeks to document concisely, 3 page tech doc, briefest code deltas possible, one page + slides withs simple diagrams.

It gets bogged down at managers managers coleads submanager for the platform team implicated. They basically say "reading the single byte at a time means its provably serial and thus has no concurrency bugs.", as indicated in my original comment.

Re: Attention Is Off By One

#180

Post by herval »

nico wrote: post #35
mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

It would be amazing if academia started replacing papers with videos + code I want to see: an explainer of the science/ideas/experiments/hipothesis And instructions on how to reproduce the experiments/results Some YouTubers are going in this direction

oh god, please, no more videos...
Post reply on HN