Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ilya Sutskever: We're moving from the age of scaling to the age of research

dwarkesh.com

Post reply on HN
Print view View original

131–140 of 374 posts

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#131

Post by slashdave »

JimmyBuckets wrote: post #106

I respect Ilya hugely as a researcher in ML and quite admire his overall humility, but I have to say I cringed quite a bit at the start of this interview when he talks about emotions, their relative complexity, and origin. Emotion is so complex, even taking all the systems in the body that it interacts with. And many mammals have very intricate socio-emotional lives - take Orcas or Elephants. There is an arrogance I…

> It's something physicists are often accused of also.

Nah. Physics is hyper-specialized. Every good physicist respects specialists.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#132

Post by orbital-decay »

>You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible.

It's definitely not small. Evolution performed a humongous amount of learning, with modern homo sapiens, an insanely complex molecular machine, as a result. We are able to learn quickly by leveraging this "pretrained" evolutionary knowledge/architecture. Same reason as why ICL has great sample efficiency.

Moreover, the community of humans created a mountain of knowledge as well, communicating, passing it over the generations, and iteratively compressing it. Everything that you can do beyond your very basic functions, from counting to quantum physics, is learned from the 100% synthetic data optimized for faster learning by that collective, massively parallel, process.

It's pretty obvious that artificially created models don't have synthetic datasets of the quality even remotely comparable to what we're able to use.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#133

Post by joshtbradley »

JimmyBuckets wrote: post #106

I respect Ilya hugely as a researcher in ML and quite admire his overall humility, but I have to say I cringed quite a bit at the start of this interview when he talks about emotions, their relative complexity, and origin. Emotion is so complex, even taking all the systems in the body that it interacts with. And many mammals have very intricate socio-emotional lives - take Orcas or Elephants. There is an arrogance I…

I think smart people across all domains fall for the trap of being overconfident in their ability to reason outside of their area of expertise. I admire those who don't, but alas we are human.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#134

Post by slashdave »

delichon wrote: post #33

If the scaling reaches the point at which the AI can do the research at all better than natural intelligence, then scaling and research amount to the same thing, for the validity of the bitter lesson. Ilya's commitment to this path is a statement that he doesn't think we're all that close to parity.

Well, he has to say that we currently aren't close to parity, because he wants people to give him money

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#135

Post by [deleted] »

JimmyBuckets wrote: post #106

I respect Ilya hugely as a researcher in ML and quite admire his overall humility, but I have to say I cringed quite a bit at the start of this interview when he talks about emotions, their relative complexity, and origin. Emotion is so complex, even taking all the systems in the body that it interacts with. And many mammals have very intricate socio-emotional lives - take Orcas or Elephants. There is an arrogance I…

[deleted]

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#136

Post by phs318u »

asolove wrote: post #91
radicaldreamer wrote: post #84

Earlier quoted context omitted.

I think the problem with that is that Grok has likely been prompted to do that in the system prompt or some prompts that get added for questions about Elon. That doesn't reflect on the actual reasoning or generalization abilities of the underlying model most likely.

Yes it does. Today on X, people are having fun baiting Grok into saying that Elon Musk is the world’s best drinker of human piss. If you hired a paid PR sycophant human, even of moderate intelligence, it would know not to generalize from “say nice things about Elon” to “say he’s the best at drinking piss”.

True. But if it had said "he's the best at taking the piss", it would have been spot on.

https://en.wikipedia.org/wiki/Taking_the_piss

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#138

Post by delichon »

tmp10423288442 wrote: post #82

He's talking his book. Doesn't mean he's wrong, but Dwarkesh is now big enough that you should assume every big name there is talking their book.

Here's a world class scientist here not because we had a hole in the schedule or he happened to be in town, but to discuss this subject that he thought and felt about so deeply that he had to write a book about it. That's a feature not a bug.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#139

Post by 827a »

newyankee wrote: post #44
gizmodo59 wrote: post #5

Even as criticism targets major model providers, his inability to answer clearly about revenue & dismissing it as a future concern reveals a great deal about today's market. It's remarkable how effortlessly he, Mira, and others secure billions, confident they can thrive in such an intensely competitive field. Without a moat defined by massive user bases, computing resources, or data, any breakthrough your researchers…

Sometimes I wonder who the rational individuals at the other end of these deals are and what makes them so confident. I always assume they have something that general public cannot deduce from public statements

There isn't necessarily rationality behind venture deals; its just a numbers game combined with the rising tide of the sector. These firms are not Berkshire. If the tide stops rising, some of the companies they invested in might actually be ok, but the venture boat sinks; the math of throwing millions at everyone hoping for one to 200x on exit does not work if the rising tide stops.

They'll say things like "we invest in people", which is true to some degree, being able to read people is roughly the only skill VCs actually need. You could probably put Sam Altman in any company on the planet and he'd grow the crap out of that company. But A16z would not give him ten billion to go grow Pepsi. This is the revealed preference intrinsic to venture; they'll say its about the people, but their choices are utterly predominated by the sector, because the sector is the predominate driver of the multiples.

"Not investing" is not an option for capital firms. Their limited partners gave them money and expect super-market returns. To those ends, there is no rationality to be found; there's just doing the best you can of a bad market. AI infrastructure investments have represented like half of all US GDP growth this year.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#140

Post by FloorEgg »

orbital-decay wrote: post #132

>You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. It's definitely not small. Evolution performed a humongous amount of learning, with modern homo sapiens, an insanely complex molecular machine, as a result. We are able to learn quickly by leveraging this "pret…

Aren't you agreeing with his point?

The process of evolution distilled down all that "humongous" amount to what is most useful. He's basically saying our current ML methods to compress data into intelligence can't compare to billions of years of evolution. Nature is better at compression than ML researchers, by a long shot.

Post reply on HN