This reveals a new source of frustration, I can't watch this in work, and I don't want to read and AI generated summary so...?
Ilya Sutskever: We're moving from the age of scaling to the age of research
31–40 of 374 posts
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#32How did Dwarkesh manage to build a brand that can attract famous people to his podcast? He didn’t have prior fame from something else in research or business, right? Curious if anyone knows his growth strategy to get here.
Seems like he’s Lex without the Rogan association so hardcore liberal folks can listen without having to buy morality offsets. He’s good, and he’s filling a void in an established underserved genre is my take.
I think its important to include that Lex is laundromat for whatever the guest is trying to sell. Dwarkesh does an impressive amount of background and speaks with experts about their expertise.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#33If the scaling reaches the point at which the AI can do the research at all better than natural intelligence, then scaling and research amount to the same thing, for the validity of the bitter lesson. Ilya's commitment to this path is a statement that he doesn't think we're all that close to parity.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#34How did Dwarkesh manage to build a brand that can attract famous people to his podcast? He didn’t have prior fame from something else in research or business, right? Curious if anyone knows his growth strategy to get here.
He's the best interviewer I ever found, try listening to his first couple episodes - they're from his dorm or something. If you can think of a similar style and originality in questioning I'd love a suggestion!
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#35The impactful innovations in AI these days aren't really from scaling models to be larger. It's more concrete to show higher benchmark scores, and this implies higher intelligence, but this higher intelligence doesn't necessarily translate to all users feeling like the model has significantly improved for their use case. Models sometimes still struggle with simple questions like counting letters in a word, and most p…
"Scaling" is going to eventually apply to the ability to run more and higher fidelity simulations such that AI can run experiments and gather data about the world as fast and as accurately as possible. Pre-training is mostly dead. The corresponding compute spend will be orders of magnitude higher.
That's true, I expect more inference time scaling and hybrid inference/training time scaling when there's continual learning rather than scaling model size or pretraining compute.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#36All coding agents are geared towards optimizing one metric, more or less, getting people to put out more tokens — or $$$.
If these agents moved towards a policy where $$$ were charged for project completion + lower ongoing code maintenance cost, moving large projects forward, _somewhat_ similar to how IT consultants charge, this would be a much better world.
Right now we have chaos monkey called AI and the poor human is doing all the cleanup. Not to mention an effing manager telling me you now "have" AI push 50 Features instead of 5 in this cycle.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#37How did Dwarkesh manage to build a brand that can attract famous people to his podcast? He didn’t have prior fame from something else in research or business, right? Curious if anyone knows his growth strategy to get here.
People are impressed by his interviews because he puts a lot of effort into researching the topic before the interview. This is a positive feedback loop.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#38How did Dwarkesh manage to build a brand that can attract famous people to his podcast? He didn’t have prior fame from something else in research or business, right? Curious if anyone knows his growth strategy to get here.
He does deep research on topics and invites people who recognize his efforts and want to engage with an informed audience.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#39So is the translation endless scaling has stopped being as effective?
It's stopped being cost-effective. Another order of magnitude of data centers? Not happening.
The business question is, what if AI works about as well as it does now for the next decade or so? No worse, maybe a little better in spots. What does the industry look like? NVidia and TSMC are telling us that price/performance isn't improving through at least 2030. Hardware is not going to save us in the near term. Major improvement has to come from better approaches.
Sutskever: "I think stalling out will look like…it will all look very similar among all the different companies. It could be something like this. I’m not sure because I think even with stalling out, I think these companies could make a stupendous revenue. Maybe not profits because they will need to work hard to differentiate each other from themselves, but revenue definitely."
Somebody didn't get the memo that the age of free money at zero interest rates is over.
The "age of research" thing reminds me too much of mid-1980s AI at Stanford, when everybody was stuck, but they weren't willing to admit it. They were hoping, against hope, that someone would come up with a breakthrough that would make it work before the house of cards fell apart.
Except this time everything costs many orders of magnitude more to research. It's not like Sutskever is proposing that everybody should go back to academia and quietly try to come up with a new idea to get things un-stuck. They want to spend SSI's market cap of $32 billion on some vague ideas involving "generalization". Timescale? "5 to 20 years".
This is a strange way to do corporate R&D when you're kind of stuck. Lots of little and medium sized projects seem more promising, along the lines of Google X. The discussion here seems to lean in the direction of one big bet.
You have to admire them for thinking big. And even if the whole thing goes bust, they probably get to keep the house and the really nice microphone holder.
Re: Ilya Sutskever: We're moving from the age of scaling to the age of research
#40So is the translation endless scaling has stopped being as effective?
Not really, but there is a finite amount of data to train models on. I found it rather interesting to hear him talk about how Gemini has been better at getting results out of the data than their competition, and how this is the first insights into a new way of dealing with how they train models on the same data to get different results.
I think the title is an interesting thing, because the scaling isn't about compute. At least as I understand it, what they're running out of is data, and one of the ways they deal with this, or may deal with this, is to have LLM's running concurrently and in competition. So you'll have thousands of models competing against eachother to solve challenges through different approaches. Which to me would suggest that the need for hardware scaling isn't about to stop.