111–120 of 347 posts
Earlier quoted context omitted.
Maybe I edited my comment too late.
> I believe the outlier problem that this solves only appears for very large models. Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.
https://arxiv.org/pdf/2208.07339.pdf
Outliers appear at model size 6.7B and are not present at 2.7B
While not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors.... In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts. I thought I discovered the error, some terms swapped in the simp…
I work at a FAANG and it was absolutely astonishing to find out how often this happens. You can make a long, impactful career by just being "the guy who adds log statements throughout the codebase and reasons through it", doing this at even a simplistic level has always shown me an astonishing fix to some long-standing issue. n.b. It also attracts a ton of political fun. People's first order reaction is denial, and i…
I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
I can't imagine judging scientific papers based on whether the author might be looking down on me, or thinks he knows better than me.
> if we were "allowed" to cite research that reads like this
Maybe you're looking down on yourself? You can cite anything you want to cite.
:D
Earlier quoted context omitted.
Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…
There was this sociologist who had written a paper for us all to read ahead of time. I started to read the damn thing, and my eyes were coming out: I couldn’t make head nor tail of it! I figured it was because I hadn’t read any of the books on the list. I had this uneasy feeling of “I’m not adequate,” until finally I said to myself “I’m gonna stop, and read one sentence slowly so I can figure out what the hell it mea…
In short, jargon matters. People here can talk about functional, procedural, and object-oriented programming because each of the three words has more than just the dictionary meaning - to those of use in the field. In the same way we can talk about linear algebra and know it doesn't mean "algebra on lines".
Yes, it's possible to write scientifically without jargon and wordiness, but it's a lot of effort and takes much more space to say "a group who follow a social structure within a society (culture, norms, values, status). They may work together to organise social life within a particular place, or they may be bound by a sense of belonging sustained across time and space"[1]
1 https://othersociologist.com/2013/11/20/sociology-of-communi...
The author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enoug…
Yeah, but it lacks the most important test: results. He hasn't actually tried it, he just thinks it will work. For such a simple change to the softmax it wouldn't take long to verify. It's really embarrassing to not do that before publishing.
The motivation follows from the same problem the author points out in the original softmax formulation that it always "forces a choice" when it may be more useful to put a "Not Applicable" option into the model itself.
https://link.springer.com/article/10.1007/s10260-021-00578-2
I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…