131–140 of 347 posts
He just says he doesn't want to spend any more time on it, which is unlikely to convince or motivate anybody else that he has discovered something important.
Earlier quoted context omitted.
Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…
There was this sociologist who had written a paper for us all to read ahead of time. I started to read the damn thing, and my eyes were coming out: I couldn’t make head nor tail of it! I figured it was because I hadn’t read any of the books on the list. I had this uneasy feeling of “I’m not adequate,” until finally I said to myself “I’m gonna stop, and read one sentence slowly so I can figure out what the hell it mea…
The criticism was """Haraway's work has been criticized for being "methodologically vague"[39] and using noticeably opaque language that is "sometimes concealing in an apparently deliberate way""""
Earlier quoted context omitted.
> I believe the outlier problem that this solves only appears for very large models. Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.
See figure 1: https://arxiv.org/pdf/2208.07339.pdf Outliers appear at model size 6.7B and are not present at 2.7B
I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
I might be missing something obvious, but I am not sure why everyone in the comments think it's a big deal. I've seen this trick in practice multiple times. For example, see this snippet from an old Google repo: https://github.com/google/flaxformer/blob/ee62754ebe5a5eeb11...
I certainly don't think it will help at all with stability. Things like Q/K layernorm are better tricks for softmax stability when scaling: https://arxiv.org/pdf/2302.05442.pdf
Earlier quoted context omitted.
He's advertising it as fixing the spiking outliers. Did your variant have those outliers beforehand?
I guess yeah I was mostly responding to Now it’s possible that softmax should be replaced wholesale, but it’s worked pretty well for the most part, except for this one wee little bug that prevents attention heads from saying nothing. So I propose a very small tweak on which I am willing to stake all future Internet claims to being correct. The tweak is so small, yet so obvious, and it’s been sitting here under everyo…
This is right below the "Have Attention Spans Been Declining? – Yes, 65%" post, lol brilliant. In general, human decreasing, AI increasing- attention.
1. Have attention spans been declining? (slimemoldtimemold.com)
338 points by janandonly 4 hours ago | flag | hide | 254 comments
2. Attention Is Off By One (evanmiller.org)
400 points by elbasti 4 hours ago | flag | hide | 129 comments
I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
But a good writer can write great articles in whatever format they wish.
Earlier quoted context omitted.
Requiring formalism does not preclude attaching an informal but intuitional description of the formal definition or proof. Unless the authors don't understand very clearly what they are talking about, or they want to prevent others from understanding their concepts too easily, I don't see why there is a reason for the authors not to attach an EIL5 in addition to formalism.
Sure. But it's an ELI5 "in addition to formalism", not "in lieu of formalism". In theory conferences like STOC or FOCS, the first section of the paper often comprises such an overview. Certainly some papers are better written than others. But sometimes a blog post cannot replace a paper, unless it also goes into the depth and detail that formalism requires. (Then it becomes a 30 page blog post, where most people don'…
You can have both and weave them together into a digestible narrative. I see Physics textbooks sometimes written this way.
Earlier quoted context omitted.
>why did no one come up with this before So it turns out someone did. Specifically google did. This exact same idea has been in flaxformers since at least November 2021. https://github.com/google/flaxformer/blame/ee62754ebe5a5eeb1... Specifically to save people a click it says: > """Softmax function with an additional virtual logit equal to zero. For compatibility with some previously trained models. This is equivale…
> I suppose only time will tell why it was ignored publicly before, maybe it doesn't do much, maybe it just fell through the cracks, maybe google just didnt push it, who knows Maybe quantization wasn't as hot back then than it is now?