Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ilya Sutskever: We're moving from the age of scaling to the age of research

dwarkesh.com

Post reply on HN
Print view View original

171–180 of 374 posts

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#171

Post by techblueberry »

Herring wrote: post #66
rockinghigh wrote: post #49

Earlier quoted context omitted.

You should read the transcript. He's including 2025 in the age of scaling. > Maybe here’s another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling. > But now the scale is so big. Is th…

Nope, Epoch.ai thinks we have enough to scale till 2030 at least. https://epoch.ai/blog/can-ai-scaling-continue-through-2030 ^ /_\ ***

Wait, nope because someone disagrees?

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#172

Post by mindwok »

Herring wrote: post #66
rockinghigh wrote: post #49

Earlier quoted context omitted.

You should read the transcript. He's including 2025 in the age of scaling. > Maybe here’s another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling. > But now the scale is so big. Is th…

Nope, Epoch.ai thinks we have enough to scale till 2030 at least. https://epoch.ai/blog/can-ai-scaling-continue-through-2030 ^ /_\ ***

That article is more about feasibility rather than desirability. There's even a section where they say:

> Settling the question of whether companies or governments will be ready to invest upwards of tens of billions of dollars in large scale training runs is ultimately outside the scope of this article.

Ilya is saying it's unlikely to be desirable, not that it isn't feasible.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#173

Post by HarHarVeryFunny »

londons_explore wrote: post #76

> These models somehow just generalize dramatically worse than people. It's a very fundamental thing My guess is we'll discover that biological intelligence is 'learning' not just from your experience, but that of thousands of ancestors. There are a few weak pointers in that direction. Eg. A father who experiences a specific fear can pass that fear to grandchildren through sperm alone. [1]. I believe this is at least…

From both an architectural and learning algorithm perspective, there is zero reason to expect an LLM to perform remotely like a brain, nor for it to generalize beyond what was necessary for it to minimize training errors. There is nothing in the loss function of an LLM to incentivize it to generalize.

However, for humans/animals the evolutionary/survival benefit of intelligence, learning from experience, is to correctly predict future action outcomes and the unfolding of external events, in a never-same-twice world. Generalization is key, as is sample efficiency. You may not get more than one or two chances to learn that life-saving lesson.

So, what evolution has given us is a learning architecture and learning algorithms that generalize well from extremely few samples.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#174

Post by samrus »

FloorEgg wrote: post #140
orbital-decay wrote: post #132

>You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. It's definitely not small. Evolution performed a humongous amount of learning, with modern homo sapiens, an insanely complex molecular machine, as a result. We are able to learn quickly by leveraging this "pret…

Aren't you agreeing with his point? The process of evolution distilled down all that "humongous" amount to what is most useful. He's basically saying our current ML methods to compress data into intelligence can't compare to billions of years of evolution. Nature is better at compression than ML researchers, by a long shot.

Sample efficiency isnt the ability to distill alot of data into good insights. Its the ability to get good insights from less data. Evolution didnt do that it had a lot of samples to get to where it did

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#175

Post by samrus »

delichon wrote: post #33

If the scaling reaches the point at which the AI can do the research at all better than natural intelligence, then scaling and research amount to the same thing, for the validity of the bitter lesson. Ilya's commitment to this path is a statement that he doesn't think we're all that close to parity.

I dont like this fanaticism around scaling. Reeks of extrapolating the s curve out to be exponential

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#176

Post by bluecheese452 »

bugglebeetle wrote: post #61
just-the-wrk wrote: post #32

Earlier quoted context omitted.

I think its important to include that Lex is laundromat for whatever the guest is trying to sell. Dwarkesh does an impressive amount of background and speaks with experts about their expertise.

His recent conversation with Sutton suggests otherwise. Friedman is a vapid charlatan par excellence. Dwarkesh suffers from a different problem, where, by rubbing shoulders with experts, he has come to the mistaken belief that he possesses expertise, absent the humility and actual work that would entail.

Spot on.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#177

Post by tintor »

jfim wrote: post #129
alyxya wrote: post #18

The impactful innovations in AI these days aren't really from scaling models to be larger. It's more concrete to show higher benchmark scores, and this implies higher intelligence, but this higher intelligence doesn't necessarily translate to all users feeling like the model has significantly improved for their use case. Models sometimes still struggle with simple questions like counting letters in a word, and most p…

Counting letters is tricky for LLMs because they operate on tokens, not letters. From the perspective of a LLM, if you ask it "this is a sentence, count the letters in it" it doesn't see a stream of characters like we do, it sees [851, 382, 261, 21872, 11, 3605, 290, 18151, 306, 480].

So what? It knows number of letters in each token, and can sum them together.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#179

Post by hiddencost »

pxc wrote: post #155

If "Era of Scaling" means "era of rapid and predictable performance improvements that easily attract investors", it sounds a lot like "AI summer". So... is "Era of Research" a euphemism for "AI winter"?

That presumes that performance improvements are necessary for commercialization.

From what I've seen the models are smart enough, what we're lacking is the understanding and frameworks necessary to use them well. We've barely scratched the surface on commercialization. I'd argue there are two things coming:

-> Era of Research -> Era of Engineering

Previous AI winters happened because we didn't have a commercially viable product, not because we weren't making progress.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#180

Post by FloorEgg »

samrus wrote: post #174
FloorEgg wrote: post #140

Earlier quoted context omitted.

Aren't you agreeing with his point? The process of evolution distilled down all that "humongous" amount to what is most useful. He's basically saying our current ML methods to compress data into intelligence can't compare to billions of years of evolution. Nature is better at compression than ML researchers, by a long shot.

Sample efficiency isnt the ability to distill alot of data into good insights. Its the ability to get good insights from less data. Evolution didnt do that it had a lot of samples to get to where it did

> Sample efficiency isnt the ability to distill alot of data into good insights

Are you claiming that I said this? Because I didn't....

There's two things going on.

One is compressing lots of data into generalizable intelligence. The other is using generalized intelligence to learn from a small amount of data.

Billions of years and all the data that goes along with it -> compressed into efficient generalized intelligence -> able to learn quickly with little data

Post reply on HN