[flagged]
[deleted]
[deleted]
Making deep learning go brrrr from first principles (2022)
11–20 of 27 posts
Re: Making deep learning go brrrr from first principles (2022)
#12[flagged]
> Experiment noise has taught me so much over several thousands of manually-run (yes, I'm a masochist, it's a personal preference) experiments converging in a few seconds.
Is there any resource that makes legible how to go about this? Or shares insights into the process of iterative learning through experiments, more broadly?
Re: Making deep learning go brrrr from first principles (2022)
#13[flagged]
[deleted]
Re: Making deep learning go brrrr from first principles (2022)
#14[flagged]
To what are you referring? Has the post been edited after your comment?
I think gp's referring to this meme [1] embedded in the article. I believe a charitable interpretation would be that it's satirising the various entities who have voiced opinions against large scale deep learning (from skeptics and theorists, to social activist types that want to slow or stop DL's march). No need to take it too seriously.
Re: Making deep learning go brrrr from first principles (2022)
#15[flagged]
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people.
It's just a meme. No need to read too much into it.
In this case, the meme came from https://gwern.net/scaling-hypothesis. I doubt that Gwern was making fun of any particular group, except perhaps those that disagree with the bitter lesson.
Re: Making deep learning go brrrr from first principles (2022)
#16[flagged]
> the author's decision to include the 'go brrr' meme
I thought it meant to involuntary shiver...
Re: Making deep learning go brrrr from first principles (2022)
#17[flagged]
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people. It's just a meme. No need to read too much into it. In this case, the meme came from https://gwern.net/scaling-hypothesis . I doubt that Gwern was making fun o…
The fluidity of language is one of the most fascinating aspects of it! English isn't English. It's much more narrowly defined with a geographic and temporal context. Meanings of words and phrases shift over time. Idioms come and origins are forgotten, but it's all folded into cultural dialects. Each generation is able to have their own mark on language, and there's probably a story told in the evolution of the English language across time that maps out the cultural and geographic changes as well. Now I'm curious what sort of research has been done into this. Regardless, the point is language changes. No one can stop it. Raging against it just reveals that you're old and out of touch. After all, I doubt those people raged against the slang of their youth. It's always the Next Generation that is The Problem.
Re: Making deep learning go brrrr from first principles (2022)
#18[flagged]
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people. There was good content in reading the article, but it turned me off at the beginning as it was rather childish -- in not-a-pleasant way -- and unfunny, and it made the rest of the article hard to read with that taste in my mouth.
I have questions...
Is the author aware of the toxic communities and their use of the meme or is the author a member of these toxic communities? Is it the authors intent to exclude you in anyway when using the meme?
And if there a bunch of "no" answers above: Why is it the authors responsibility to keep track with your associations with this meme?
Re: Making deep learning go brrrr from first principles (2022)
#19[flagged]
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people. There was good content in reading the article, but it turned me off at the beginning as it was rather childish -- in not-a-pleasant way -- and unfunny, and it…
I did address those earlier in terms of association, I'm not sure if I understand the aggressive tone of this response however, or of a few of the assumptions in it. A few other people have made similar points about this culture as well.
Re: Making deep learning go brrrr from first principles (2022)
#20[flagged]
> Experiment noise has taught me so much over several thousands of manually-run (yes, I'm a masochist, it's a personal preference) experiments converging in a few seconds. Is there any resource that makes legible how to go about this? Or shares insights into the process of iterative learning through experiments, more broadly?
I'd like to create it in the future, but I can share a rough version of what I have. Basically, as long as you gain more information than noise by scaling your experiments down (i.e., the smaller models create similar directions in the needle), then it's easier to run lots of experiments, and much more cheaply oftentimes too. Remember that performance oftentimes runs on a log(1+p) curve in terms of cost/time/complexity vs reward.
If you write your codebases to minimize the expected time to make a particular change (fewer files, simpler proxy problem [as long as it transfers to your larger-scale problem!], more 'flat' structure, simple dataloaders, etc), then that is extremely valuable as well. What you are optimizing is the average time from idea-to-answer.
What is nice is that experiments go from strongly-planned, biased (from human beliefs, etc) things to a more high-variance process where there isn't as much requirement to double-and-triple check everything before running it. There is often an _extremely_ poor mismatch between the very first impression of what many of us think will work and will not and what actually works when the dust settles. Sorta like how SGD bounces around the loss landscape before settling in.
One thing that also is easier with fast experiments is to make the 'opposite' change if a planned change doesn't work out well, just to see how it goes. A tiny thing, but if you're going sequentially instead of batched (in terms of batched experiments) it can be useful.
I use as few libraries as possible. A lot of the bulky monitoring solutions can be good for some enterprise things, I don't find they're good for innovating, but instead, integrating. You innovate on small test problems that are appropriately representative. When done, you integrate and see what difference it makes.
Doing dumb things is one of the best ways to learn. Having access to everything to be able to print it out/log it/look at it in a chart is great too.
But everything is bottlenecked by speed. Increase your experiment speed, where (critically!) your speed includes the time from idea to first answer coming back (and not just plain ol' runtime), and you'll find oftentimes it's easy to get 5-10x research speed improvements. We just don't do research all that efficiently these days, I don't think. But we can. And I'm sure we will, eventually. :3
There are a few things like scaling that can be difficult, but math is still math, and if you pick your proxies right, it should be an explored-translation-step after the proxy problem behaves as well as it is reasonably able. And if you have a secondary, slightly larger proxy problem, that's an even better thing in my experience, since it prevents the sticker shock of broken graphs and things failing for an unknown reason. All about how well you can pass the peach from one person to another. :')))) <3 <3 :')))) :'D :'D