Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ilya Sutskever: We're moving from the age of scaling to the age of research

dwarkesh.com

Post reply on HN
Print view View original

341–350 of 374 posts

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#341

Post by Animats »

tim333 wrote: post #330
Animats wrote: post #39

Earlier quoted context omitted.

It's stopped being cost-effective. Another order of magnitude of data centers? Not happening. The business question is, what if AI works about as well as it does now for the next decade or so? No worse, maybe a little better in spots. What does the industry look like? NVidia and TSMC are telling us that price/performance isn't improving through at least 2030. Hardware is not going to save us in the near term. Major i…

A difference with mid-1980s AI is the hardware is way more capable now so even flawed algorithms can do quite economically significant stuff like Claude Code etc. Recent headline "Anthropic projects as much as $26 billion in annualized revenue in 2026". With that sort of revenue you'd expect some significant spend on R&D.

> "Anthropic projects as much as $26 billion in annualized revenue in 2026".

Anthropic projects a lot. It's hard to get actuals from Anthropic.[1] They're privately held, so they don't have to report actuals publicly. [1] says "Anthropic has, through July 2025, made around $1.5 billion in revenue." $26 billion for 2026 seems unlikely.

This is revenue, not profit.

[1] https://www.wheresyoured.at/howmuchmoney/

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#342

Post by Animats »

sd9 wrote: post #276
giardini wrote: post #228

Earlier quoted context omitted.

What kind of ideas would be intellectual property that was not shared? Isn't every part of LLMs, except the order of processes, publicly known ? Is there some magic algorithm previously unrevealed and held secret by a cabal of insiders?

Why are some models better than others today if everything is publicly known and many organisations have access to massive resources? Somebody has to come up with an idea first. Before they share it, it is not publicly known. Ilya has previously come up with plenty of productive ideas. I don't think it's a stretch to think that he has some IP that is not publicly known. Even seemingly simple things like how you shuff…

> Somebody has to come up with an idea first.

There are lots of ideas. Some may work.

The space in which people seem to be looking is deep learning on something other than text tokens. Yet most successes punt on feature extraction / "early vision" and just throw compute at raw pixels. That's the "bitter lesson" approach, which seems to be hitting the ceiling of how many gigawatts of data center you can afford.

Is there a useful non-linguistic abstraction of the real world that works and leads to "common sense"? Squirrels must have something; they're not verbal and have a brain the size of a peanut. But what?

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#343

Post by qcnguy »

gnfargbl wrote: post #284
gizmo wrote: post #275

Earlier quoted context omitted.

You describe the "fake email jobs" theory of employment. Given that there are way fewer email jobs in China does this imply that China will benefit more from AI? I think it might.

Are there fewer busy-work jobs in China? If so, why? It's an interesting assertion, but human nature tends to be universal.

Could argue there are more. Lots of loss making SOEs in China.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#344

Post by zeroonetwothree »

otabdeveloper4 wrote: post #235
joelthelion wrote: post #227

> When do you expect that impact? I think the models seem smarter than their economic impact would imply. > Yeah. This is one of the very confusing things about the models right now. As someone who's been integrating "AI" and algorithms into people's workflows for twenty years, the answer is actually simple. It takes time to figure out how exactly to use these tools, and integrate them into existing tooling and workf…

> the models seem smarter than their economic impact would imply Key word is "seem".

Kind of like how some humans seem smart during the interview but then are incapable of actually doing anything properly.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#345

Post by __loam »

omnimus wrote: post #254
ACCount37 wrote: post #246

Earlier quoted context omitted.

That's like saying that a modern calculator and a mechanical arithmometer have very little in common. Sure, the parts are all different, and the construction isn't even remotely similar. They just happen to be doing the same thing.

But they just don't happen to be doing the same thing. People claiming otherwise have to first prove that we are comparing the same thing. This whole strand of “inteligence is just a compression” may be possible but it's just as likely (if not a massively more likely) that compression is just a small piece or even not at all how biological inteligence works. In your analogy it's more like comparing modern calculator…

Well put and captures my feelings on this

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#346

Post by ilaksh »

anthonypasq wrote: post #339
ilaksh wrote: post #156

Earlier quoted context omitted.

The question of how emotions function and how they might be related to value functions is absolutely central to that discussion and very relevant to his field. Doing fundamental AI research definitely involves adjacent fields like neurobiology etc. Re: the discussion, emotions actually often involve high level cognition -- it's just subconscious. Let's take a few examples: - amusement: this could be something simple…

i think the contention is the idea that emotions are simple.

Yes, that is what they were suggesting in the interview, which I think is not quite accurate, so I replied with the comment above.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#347

Post by twodave »

I’d settle for the “age of being able to point the LLM at the entire codebase, describe a new feature, and see it implemented based on the patterns and idioms already present in that codebase”. My impression is the only thing between here and there is context size.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#348

Post by giardini »

snapcaster wrote: post #317
giardini wrote: post #232

Earlier quoted context omitted.

I'll be convinced LLMs are a reasonable approach to AI when an LLM can give reasonable answers after being trained with approximately the same books and classes in school that I was once I completed my college education.

Why do you think this standard you're applying is reasonable or meaningful?

For the same reason anyone would: if an AI can reason to a human level after having been educated in a manner similar to a human then it is likely that we [the educators] have captured something akin to human intelligence.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#349

Post by jonmc12 »

JimmyBuckets wrote: post #106

I respect Ilya hugely as a researcher in ML and quite admire his overall humility, but I have to say I cringed quite a bit at the start of this interview when he talks about emotions, their relative complexity, and origin. Emotion is so complex, even taking all the systems in the body that it interacts with. And many mammals have very intricate socio-emotional lives - take Orcas or Elephants. There is an arrogance I…

I don't think trans-disciplinary inquiry is arrogance - the intellectual fields are somewhat arbitrary relative to how human expertise relates to real world problems. But, effective trans-disciplinary inquiry requires awareness of philosophical commitments, and familiarity with existing literature/theory.

The bigger challenge might be that people with ML expertise need to solve problems of human-AI interaction and alignment because the training for the former is uni-disciplanary while the latter is trans-disciplinary.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#350

Post by ojbyrne »

latexr wrote: post #294
ojbyrne wrote: post #190

Earlier quoted context omitted.

I’m curious how you deduced it’s from 2024. Timestamps on the article and the embedded video are both November 2025.

It says at the top it was published Aug 20, 2024, and the Internet Archive has it since Nov 13, 2024. https://web.archive.org/web/20241113185615/https://epoch.ai/...

Sorry, I didn't follow the thread, thought you were referring to the top level article.
Post reply on HN