Attention, scaling laws, diffusion, vision transformers, Bert/Roberta, CLIP, chinchilla, chatgpt-related papers, nerf, flamingo, RETRO/some retrieval sota
[deleted]
Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
71–80 of 144 posts
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#72Attention, scaling laws, diffusion, vision transformers, Bert/Roberta, CLIP, chinchilla, chatgpt-related papers, nerf, flamingo, RETRO/some retrieval sota
what do you mean 'scaling laws'?
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#73What about just asking Carmack on twitter?
Or, more directly, ask Sutskever...
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#74A lot of other posts here are biased to recent papers, and papers that had "a big impact", but miss a lot of foundations. I think this reddit post on the most foundational ML papers gives a lot more balanced overview: https://www.reddit.com/r/MachineLearning/comments/zetvmd/d_i...
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#75Earlier quoted context omitted.
>Will receive Turing Award This is the weird thing - hopefully not! Hopefully there's even better NN models coming out every 5-10 years and we look back on transformers as 'just a phase' sort of like how we look back at RNN's (which were no less of an amazing achievement - look at the proliferation of voice assistants), as potentially obsolete technology today. Fore example, attention is great and does a really good…
> symbology Off-topic, but now I have William Dafoe going "What's the 'symbology' here? The symbolism ..." in my head (from Boondock Saints).
Even thou I watched that movie 20 years ago. I will never forget that scene.
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#76Earlier quoted context omitted.
I did, without success.
I asked him too. He said: - Who are you, and how did you get into my house?
I wouldn’t advise this after seeing what Carmack did to that guy he got in a headlock. [0]
“That was the tap part”, makes me laugh every time.
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#77Carmack says he's pursuing a different path to AGI, then goes straight to the guy at the center of the most saturated area of machine learning (deep learning)? I would've hoped he'd be exploring weirder alternatives off the beaten path. I mean, neural networks might not even be necessary for AGI, but no one at OpenAI is going to tell Carmack that.
The most off the beaten path to AGI I heard through the grapevine is to not have artificial neural networks, as in algorithms involving matmul running on silicon, at all. But instead, going on the path of the laziest engineer is the best engineer, to rely on the fact that neurons, actual neurons from someone's brain, already "know" how to make efficient, good-enough, general learning architectures and therefore in order to obtain programmatic human-like intelligence one would 'simply'† have to implant them not in mice [1] but in an actual vat and 'simply' interface with the whatever a group of neurons can be called, a soma(?). Given this Brain-on-a-Chip architecture, we wouldn't have to stick GPUs in our cars to achieve self-driving, but even more wetware (and of course, ignore the occasional screams of dread as the wetware becomes aware of themselves and how condemned they are to an existence of left-right-accelerate-break).
It would have been interesting seeing someone like Carmack going in this direction, but from the little details he gave he seems less interested in cells and Kjeldahl flasks and more of the same type-a-type-a on the ol' QWERTY.
† 'simply' might involve multiple decades of research and Buffett knows how many billions
[1] Human neurons implanted in mice influence behavior, https://www.nature.com/articles/s41586-022-05277-w
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#78(Partly copied from https://news.ycombinator.com/item?id=34640251 .) On models: Obviously, almost everything is Transformer nowadays (Attention is all you need paper). However, I think to get into the field, to get a good overview, you should also look a bit beyond the Transformer. E.g. RNNs/LSTMs are still a must learn, even though Transformers might be better in many tasks. And then all those memory-augmented model…
Thanks for sharing. Cool to see someone from Aachen NLP group. I'll be visiting Aachen/Düsseldorf/Heidelberg area in spring. Do you know of any local ML meetups open to general (ML engineer/programmer) public?
Unfortunately, not really. We used to have some RWTH internal meetups, although that has been somewhat interrupted since Corona, and not really recovered afterwards.
Aachen has quite a few companies with activity on NLP or speech recognition, mostly due to my professor Hermann Ney. E.g. there is Apple, Amazon, Nuance, eBay. And lesser-known AppTek. And in Cologne, you have DeepL. In all those companies, you find many people from our group. And then, at the RWTH Aachen University, you have our NLP/speech group, and also the computer vision group.
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#79(Partly copied from https://news.ycombinator.com/item?id=34640251 .) On models: Obviously, almost everything is Transformer nowadays (Attention is all you need paper). However, I think to get into the field, to get a good overview, you should also look a bit beyond the Transformer. E.g. RNNs/LSTMs are still a must learn, even though Transformers might be better in many tasks. And then all those memory-augmented model…
I do wonder whether people behind Attention is all you need paper Will receive Turing Award It is being cited often
The authors did not really expect it to be such a huge influence. You could also argue, it is a somewhat natural next step. This paper did not invent self-attention nor attention. Attention was already very popular, specifically for machine translation, and a few other papers already did use self-attention at that point in time. It was just the first paper which solely used attention and self-attention and nothing else.
Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?
#80Carmack says he's pursuing a different path to AGI, then goes straight to the guy at the center of the most saturated area of machine learning (deep learning)? I would've hoped he'd be exploring weirder alternatives off the beaten path. I mean, neural networks might not even be necessary for AGI, but no one at OpenAI is going to tell Carmack that.
What a waste it would be to think you are pursuing a different path only to discover you spent a year reinventing something that you could have learned by reading papers for a few days.