Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ilya Sutskever: We're moving from the age of scaling to the age of research

dwarkesh.com

Post reply on HN
Print view View original

321–330 of 374 posts

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#321

Post by HarHarVeryFunny »

joelthelion wrote: post #227

> When do you expect that impact? I think the models seem smarter than their economic impact would imply. > Yeah. This is one of the very confusing things about the models right now. As someone who's been integrating "AI" and algorithms into people's workflows for twenty years, the answer is actually simple. It takes time to figure out how exactly to use these tools, and integrate them into existing tooling and workf…

No doubt LLMs and tooling will continue to improve, and best use cases for them better understood, but what Ilya seems to be referring to is the massive disconnect between the headline-grabbing benchmarks such as "AI performs at PhD level on math", etc, and the real-world stupidity of these models such as his example of a coding agent toggling between generating bug #1 vs bug #2, which in fact largely explains why the current economic and visible impact is much less than if the "AI is PhD level" benchmark narrative was actually true.

Calling LLMs "AI" makes them sound much more futuristic and capable than they actually are, and being such a meaningless term invites extrapolation to equally meaningless terms like AGI and visions of human-level capability.

Let's call LLMs what they are - language models - tools for language-based task automation.

Of course we eventually will do this. Fuzzy meaningless names like AI/AGI will always be reserved for the cutting edge technology du jour, and older tech that is realized in hindsight to be much more limited will revert to being called by more specific names such as "expert system", "language model", etc.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#322

Post by ACCount37 »

filleduchaos wrote: post #320
ACCount37 wrote: post #313

Earlier quoted context omitted.

> Otherwise they are simply two completely different objects. That's where you're wrong. Both objects reflect the same mathematical operations in their structure. Even if those were inscrutable alien artifacts to you, even if you knew nothing about who constructed them, how or why? If you studied them, you would be able to see the similarities laid bare. Their inputs align, their outputs align. And if you dug deep en…

> That's where you're wrong. Both objects reflect the same mathematical operations in their structure. This is missing the point by a country mile, I think. All navel-gazing aside, understanding every bit of how an arithmometer works - hell, even being able to build one yourself - tells you absolutely nothing about how the Z80 chip in a TI-83 calculator actually works. Even if you take it down to individual component…

You don't get it at all, do you?

"Implements the same math" IS the similarity.

I'm baffled that someone in CS, a field ruled by applied abstraction, has to be explained over and over again that abstraction is a thing that exists.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#323

Post by firestell »

bossyTeacher wrote: post #302
hn_acc1 wrote: post #158

Earlier quoted context omitted.

You think that you will be ALLOWED to continue to use AI for free once it can create a LOT of wealth? Or will you have to pay royalties? The rich CEOs don't want MORE competition - they want LESS competition for being rich. I'm sure they'll find a way to add a "any vibe-coded business owes us 25% royalties" clause any day now, once the first big idea makes some $$. If that ever happens. They're NOT trying to liberate…

This. This is what I find hilarious that even smart HN folks seem unable to understand. Transformers tech products are a service offered by private companies who are under no obligation to serve it to you indefinitely. At any given point, they are free to end public access. And you better believe that they will do so if it is in their interest. inb4 open source models, those models are also hosted on the servers of p…

Thats borderline aluminum hat conspiracy theory. Corporations arent a monolith, you think amazon is ever going to stop you from renting machines so that you cant run your AI models instead of buying from OpenAI? They have no horse in that race.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#324

Post by Jeff_Brown »

alex43578 wrote: post #264
giardini wrote: post #232

Earlier quoted context omitted.

I'll be convinced LLMs are a reasonable approach to AI when an LLM can give reasonable answers after being trained with approximately the same books and classes in school that I was once I completed my college education.

I'll be convinced cars are a reasonable approach to transportation when it can take me as far as a horse can on a bale of hay.

That is such a beautiful analogy that now I will read your other comments.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#325

Post by Version467 »

just-the-wrk wrote: post #38
SilverElfin wrote: post #7

How did Dwarkesh manage to build a brand that can attract famous people to his podcast? He didn’t have prior fame from something else in research or business, right? Curious if anyone knows his growth strategy to get here.

He does deep research on topics and invites people who recognize his efforts and want to engage with an informed audience.

That, plus he's quick enough to come up with good follow-up questions on the spot. It's so frustrating listening to interviews where the interviewer simply glosses over interesting/controversial statements because they either don't care, or don't know enough to identify a statement as controversial. In contrast, Dwarkesh is incredible at this. 9/10 times when I'm confused about a statement that a guest makes on his show he will immediately follow up by asking for clarification or pushing back. It's so refreshing.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#326

Post by jebarker »

HarHarVeryFunny wrote: post #308
jebarker wrote: post #201

Earlier quoted context omitted.

> what evolution has given us is a learning architecture and learning algorithms that generalize well from extremely few samples. This sounds magical though. My bet is that either the samples aren’t as few as they appear because humans actually operate in a constrained world where they see the same patterns repeat very many times if you use the correct similarity measures. Or, the learning that the brain does during…

> This sounds magical though Not really, this is just the way that evolution works - survival of the fittest (in the prevailing environment). Given that the world is never same twice, then generalization is a must-have. The second time you see the tiger charging out, you better have learnt your lesson from the first time, even if everything other than "it's a tiger charging out" is different, else it wouldn't be very…

> Note how different, and massively more complex, the spatio-temporal real world of messy analog never-same-twice dynamics is to the 1-D symbolic/discrete world of text that "AI" is currently working on.

I agree that the real world perceived by a human is vastly more complex than a sequence of text tokens. But it’s not obvious to me that it’s actually less full of repeating patterns or that learning to recognize and interpolate those patterns (like an LLM does) is insufficient for impressive generalization. I think it’s too hard to reason about this stuff when the representations in LLMs and the brain are so high-dimensional.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#327

Post by _giorgio_ »

JohnnyMarcone wrote: post #88
_giorgio_ wrote: post #63

Scaling is not over, there's no wall. Oriol Vinyals VP of Gemini research https://x.com/OriolVinyalsML/status/1990854455802343680?t=oC...

He didn't say it's over, just that continued scaling won't be transformational.

Oriol Vinyals said that.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#328

Post by filleduchaos »

ACCount37 wrote: post #322
filleduchaos wrote: post #320

Earlier quoted context omitted.

> That's where you're wrong. Both objects reflect the same mathematical operations in their structure. This is missing the point by a country mile, I think. All navel-gazing aside, understanding every bit of how an arithmometer works - hell, even being able to build one yourself - tells you absolutely nothing about how the Z80 chip in a TI-83 calculator actually works. Even if you take it down to individual component…

You don't get it at all, do you? "Implements the same math" IS the similarity. I'm baffled that someone in CS, a field ruled by applied abstraction, has to be explained over and over again that abstraction is a thing that exists.

In case you have missed it in the middle of the navel-gazing about abstraction, this all started with the comment "Please stop comparing these things to biological systems. They have very little in common."[0]

If you insist on continuing to miss the point even when told explicitly that the comment is referring to what's inside the box, not its interface, then be my guest. There isn't much of a sensible discussion about engineering to be had with someone who thinks that e.g. the sentence "Please stop comparing [nuclear reactors] to [coal power plants]. They have very little in common" can be countered with "but abstraction! they both produce electricity!".

For the record, I am not the one you have been replying to.

[0] https://news.ycombinator.com/item?id=46053563

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#329

Post by mountainriver »

pxc wrote: post #155

If "Era of Scaling" means "era of rapid and predictable performance improvements that easily attract investors", it sounds a lot like "AI summer". So... is "Era of Research" a euphemism for "AI winter"?

Take it with a grain of salt, this is one man’s opinion, even though he is a very smart man.

People have been screaming about an AI winter since 2010 and it never happened, it certainly won’t happen now that we are close to AGI which is a necessity for national defense.

I prefer Dario’s perspective here, which is that we’ve seen this story before in deep learning. We hit walls and then found ways around them with better activation functions, regularization and initialization.

This stuff is always a progression in which we hit roadblocks and find ways around them. The chart of improvement is still linearly up and to the right. Those gains are the cumulation of small improvements adding up.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#330

Post by tim333 »

Animats wrote: post #39
andy_ppp wrote: post #2

So is the translation endless scaling has stopped being as effective?

It's stopped being cost-effective. Another order of magnitude of data centers? Not happening. The business question is, what if AI works about as well as it does now for the next decade or so? No worse, maybe a little better in spots. What does the industry look like? NVidia and TSMC are telling us that price/performance isn't improving through at least 2030. Hardware is not going to save us in the near term. Major i…

A difference with mid-1980s AI is the hardware is way more capable now so even flawed algorithms can do quite economically significant stuff like Claude Code etc. Recent headline "Anthropic projects as much as $26 billion in annualized revenue in 2026". With that sort of revenue you'd expect some significant spend on R&D.
Post reply on HN