Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 144 posts

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#121

Post by stuntkite »

pavon wrote: post #80
optimalsolver wrote: post #17

Carmack says he's pursuing a different path to AGI, then goes straight to the guy at the center of the most saturated area of machine learning (deep learning)? I would've hoped he'd be exploring weirder alternatives off the beaten path. I mean, neural networks might not even be necessary for AGI, but no one at OpenAI is going to tell Carmack that.

What a waste it would be to think you are pursuing a different path only to discover you spent a year reinventing something that you could have learned by reading papers for a few days.

That's like a constant cycle for me. The stuff that grows from it is the things that keep growing and sticking around and I don't find any other literature directly replacing it or enhancing it. When I do find things that replace a bunch of my work I'm thrilled because I don't have to do that now and I can focus my energy on the other threads. Every once in a while I get competitive and it hurts, but if I'm being honest if I find something that gets me that way I've got a special appreciation for that moment.

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#122

Post by loveparade »

chrgy wrote: post #50

From ChatGPT, although personally I think this list is bit old but should be at the 60% mark at the very least Deep Learning: AlexNet (2012) VGGNet (2014) ResNet (2015) GoogleNet (2015) Transformer (2017) Reinforcement Learning: Q-Learning (Watkins & Dayan, 1992) SARSA (R. S. Sutton & Barto, 1998) DQN (Mnih et al., 2013) A3C (Mnih et al., 2016) PPO (Schulman et al., 2017) Natural Language Processing: Word2Vec (Mikolo…

You are getting downvoted because this list if from ChatGPT, but as a researcher in the field, this list is actually really good, except for perhaps the SARSA and GLUE papers, which are less generally relevant. I would add WaveNet, the Seq2Seq paper, GANs, some optimizer papers (e.g. Adam), diffusion models, and some of the newer Transformer variants.

I'm very confident that this is pretty much what any researcher, including Ilya, would recommend. It really isn't hard to find those resources, they are simply the most cited papers. Of course you can go deeper into any of the subfields if you desire.

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#123

Post by [deleted] »

bathtub365 wrote: post #107
klabb3 wrote: post #94

Earlier quoted context omitted.

I think there’s broad consensus that slavery only applies to human labor. Even within that spectrum people avoid the term (see forced prison labor). We also don’t use it for animal labor, for instance.

Human slaves were often considered to be less than human or, at the very least, not deserving of basic rights that other humans enjoyed, as part of the moral and ethical frameworks that supported the practice. I think we might see the same shift in dominant ideology if we do have “true” AGI. I’m sure I could be convinced that an intelligence that develops and grows over a number of years begins to have a right to exi…

[deleted]

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#124

Post by optimalsolver »

93po wrote: post #118
ramraj07 wrote: post #87

Earlier quoted context omitted.

This is pretty much the same deal in biology as well. At calico, at verily, at CZI, even at Allen, same story - they say they will reinvent biology research and then go get the same narrow minded professors and CEOs who run the status quo and end up as one more of the same stuff. Neuralink is the only place where this pattern seemed to break a bit but then seems like Elon came into his own path with trying to push fo…

> breaking basic ethics This didn't happen

https://fortune.com/2022/12/13/elon-musk-neuralink-animal-ex...

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#125

Post by PartiallyTyped »

hardware2win wrote: post #6
albertzeyer wrote: post #2

(Partly copied from https://news.ycombinator.com/item?id=34640251 .) On models: Obviously, almost everything is Transformer nowadays (Attention is all you need paper). However, I think to get into the field, to get a good overview, you should also look a bit beyond the Transformer. E.g. RNNs/LSTMs are still a must learn, even though Transformers might be better in many tasks. And then all those memory-augmented model…

I do wonder whether people behind Attention is all you need paper Will receive Turing Award It is being cited often

Attention existed before that paper and was incorporated to LSTMs until that point in time.

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#126

Post by zo1 »

aj7 wrote: post #98
sho_hn wrote: post #90

> "You’ll find people who can wax rhapsodic about the singularity and how everything is going to change with AGI. But if I just look at it and say, if 10 years from now, we have ‘universal remote employees’ that are artificial general intelligences, run on clouds, and people can just dial up and say, ‘I want five Franks today and 10 Amys, and we’re going to deploy them on these jobs,’ and you could just spin up like…

Computer time is paid for.

What happens when we digitize ourselves and can run said snapshot image on "computer time"? We can barely cope with legal issues in the digital age now.

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#127

Post by TimPC »

ilaksh wrote: post #113
TimPC wrote: post #60

Earlier quoted context omitted.

Why do you think multimodal transformers will get us anywhere near general purpose AI? Multimodal transformers are basically a technology for sequence-to-sequence intelligent mappings and it seems to me extremely unlikely that general intelligence is one or more specific sequence-to-sequence mappings. Many specific purpose problems are sequence-to-sequence but these tend to be specialized functionalities operating in…

A couple of things. 1) As I said, many people have different ideas of what we are talking about. I assume that for you general purpose AI has more capabilities, such as the ability to quickly learn tasks to a high level on the fly. For me, it still qualifies as general purpose if it can do most tasks but relies on a lot of pre-training and let's say knowledgebase look up. 2) It seems obvious to me that ChatGPT proves…

For 1) I agree but ChatGPT is a specific purpose sequence to sequence model. It’s fairly obvious to me it’s not general purpose and it even fails sometimes at correctly reading content it generates. It also doesn’t understand correctness and often ends up generating incorrect content. Our best example of this not being general purpose is how staggeringly bad ChatGPT is at math which is blatantly obvious when you think about how it is designed.

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#128

Post by alan-stark »

albertzeyer wrote: post #78
alan-stark wrote: post #65

Earlier quoted context omitted.

Thanks for sharing. Cool to see someone from Aachen NLP group. I'll be visiting Aachen/Düsseldorf/Heidelberg area in spring. Do you know of any local ML meetups open to general (ML engineer/programmer) public?

Unfortunately, not really. We used to have some RWTH internal meetups, although that has been somewhat interrupted since Corona, and not really recovered afterwards. Aachen has quite a few companies with activity on NLP or speech recognition, mostly due to my professor Hermann Ney. E.g. there is Apple, Amazon, Nuance, eBay. And lesser-known AppTek. And in Cologne, you have DeepL. In all those companies, you find many…

Sounds like an "NLP valley" with Prof. Ney as Aachen's own Fred Terman :)

Re: Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack?

#130

Post by mellosouls »

sillysaurusx wrote: post #86

"The email including them got lost to Meta's two-year auto-delete policy by the time I went back to look for it last year. I have a binder with a lot of them printed out, but not all of them." RIP. If it's any consolation, it sounds like the list is at least three years old by now. Which is a long time considering that 2016 is generally regarded as the date of the deep learning revolution.

Sorry - where is that sourced from? Or are you meaning it was a personal communication to you? Or it's a joke?
Post reply on HN