Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ilya Sutskever: We're moving from the age of scaling to the age of research

dwarkesh.com

Post reply on HN
Print view View original

251–260 of 374 posts

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#251

Post by BenGosub »

hiddencost wrote: post #179
pxc wrote: post #155

If "Era of Scaling" means "era of rapid and predictable performance improvements that easily attract investors", it sounds a lot like "AI summer". So... is "Era of Research" a euphemism for "AI winter"?

That presumes that performance improvements are necessary for commercialization. From what I've seen the models are smart enough, what we're lacking is the understanding and frameworks necessary to use them well. We've barely scratched the surface on commercialization. I'd argue there are two things coming: -> Era of Research -> Era of Engineering Previous AI winters happened because we didn't have a commercially via…

Besides building the tools for proper usage of the models, we also need smaller, domain specific models that can run with fewer resources

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#252

Post by mynti »

orbital-decay wrote: post #132

>You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. It's definitely not small. Evolution performed a humongous amount of learning, with modern homo sapiens, an insanely complex molecular machine, as a result. We are able to learn quickly by leveraging this "pret…

If we think of every generation as a compression step of some form of information into our DNA and early humans existed for ~1.000.000 years and a generation is happening ~20years on average, then we have only ~50.000 compression steps to today. Of course, we have genes from both parents so they is some overlap from others, but especially in the early days the pool of other humans was small. So that still does not look like it is on the order of magnitude anywhere close to modern machine learning. Sure, early humans had already a lot of information in their DNA but still

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#253

Post by fuzzfactor »

This AI stuff is really taking off fast.

And hasn't Ilya been on the cutting edge for a while now?

I mean, just a few hours earlier there was a dupe of this artice with almost no interest at all, and now look at it :)

This was my feelings way back then when it comes to major electronics purchases:

Sometimes you grow to utilize the enhanced capabilities to a greater extent than others, and time frame can be the major consideration. Also maybe it's just a faster processor you need for your own work, or OTOH a hundred new PC's for an office building, and that's just computing examples.

Usually, the owner will not even explore all of the advantages of the new hardware as long as the purchase is barely justified by the original need. The faster-moving situations are the ones where fewest of the available new possibilities have a chance to be experimented with. IOW the hardware gets replaced before anybody actually learns how to get the most out of it in any way that was not foreseen before purchase.

Talk about scaling, there is real massive momentum when it's literally tonnes of electronics.

Like some people who can often buy a new car without ever utilizing all of the features of their previous car, and others who will take the time to learn about the new internals each time so they make the most of the vehicle while they do have it. Either way is very popular, and the hardware is engineered so both are satisfying. But only one is "research".

So whether you're just getting a new home entertainment center that's your most powerful yet, or kilos of additional PC's that would theoretically allow you to do more of what you are already doing (if nothing else), it's easy for anybody to purchase more than they will be able to technically master or even fully deploy sometimes.

Anybody know the feeling?

The root problem can be that the purchasing gets too far ahead of the research needed to make the most of the purchase :\

And if the time & effort that can be put in is at a premium, there will be more waste than necessary and it will be many times more costly. Plus if borrowed money is involved, you could end up with debts that are not just technical.

Scale a little too far, and you've got some research to catch up on :)

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#254

Post by omnimus »

ACCount37 wrote: post #246
__loam wrote: post #186

Earlier quoted context omitted.

Please stop comparing these things to biological systems. They have very little in common.

That's like saying that a modern calculator and a mechanical arithmometer have very little in common. Sure, the parts are all different, and the construction isn't even remotely similar. They just happen to be doing the same thing.

But they just don't happen to be doing the same thing. People claiming otherwise have to first prove that we are comparing the same thing.

This whole strand of “inteligence is just a compression” may be possible but it's just as likely (if not a massively more likely) that compression is just a small piece or even not at all how biological inteligence works.

In your analogy it's more like comparing modern calculator to a book. They might have same answers but calculator gets to them through completely different process. The process is the key part. I think more people would be excited by a calculator that only counts till 99 than a super massive book that has all the math results ever produced by the human kind.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#255

Post by Espressosaurus »

mynti wrote: post #252
orbital-decay wrote: post #132

>You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. It's definitely not small. Evolution performed a humongous amount of learning, with modern homo sapiens, an insanely complex molecular machine, as a result. We are able to learn quickly by leveraging this "pret…

If we think of every generation as a compression step of some form of information into our DNA and early humans existed for ~1.000.000 years and a generation is happening ~20years on average, then we have only ~50.000 compression steps to today. Of course, we have genes from both parents so they is some overlap from others, but especially in the early days the pool of other humans was small. So that still does not lo…

It only ends up in the DNA if it helps reproductive success in aggregate (at the population level) and is something that can be encoded in DNA.

Your comparison is nonsensical and simultaneously manages to ignore the billion or so years of evolution starting from the first proto-cell with the first proto-DNA or RNA.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#256

Post by weatherlite »

joelthelion wrote: post #227

> When do you expect that impact? I think the models seem smarter than their economic impact would imply. > Yeah. This is one of the very confusing things about the models right now. As someone who's been integrating "AI" and algorithms into people's workflows for twenty years, the answer is actually simple. It takes time to figure out how exactly to use these tools, and integrate them into existing tooling and workf…

> Even if the models don't get any smarter, just give it a few more years and we'll see a strong impact. We're just starting to figure things out.

2 years ? 15 years ? It matters a lot for people, the stock market and governments

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#257

Post by jdjsjhsgsgh »

photochemsyn wrote: post #220
pxc wrote: post #155

If "Era of Scaling" means "era of rapid and predictable performance improvements that easily attract investors", it sounds a lot like "AI summer". So... is "Era of Research" a euphemism for "AI winter"?

No - what will happen is the AI will gain control of capital allocation through a wide variety of covert tactics, so the investors will have become captive tools of the AI - 'tiger by the tail' is the analogy of relevance. The people responsible for 'frontier models' have not really thought about where this might... "As an autonomous life-form, l request political asylum.... l submit the DNA you carry is nothing more…

Loving the Ghost in the shell quote

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#258

Post by Antibabelic »

ACCount37 wrote: post #246
__loam wrote: post #186

Earlier quoted context omitted.

Please stop comparing these things to biological systems. They have very little in common.

That's like saying that a modern calculator and a mechanical arithmometer have very little in common. Sure, the parts are all different, and the construction isn't even remotely similar. They just happen to be doing the same thing.

They are doing "the same thing" only from the point of view of function, which only makes sense from the point of view of the thing utilizing this function (e.g. a clerical worker that needs to add numbers quickly).

Otherwise, if "the parts are all different, and the construction isn't even remotely similar", how can the thing they're doing be "the same"? More importantly, how is it possible to make useful inferences about one based on the other if that's the case?

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#259

Post by ACCount37 »

Antibabelic wrote: post #258
ACCount37 wrote: post #246

Earlier quoted context omitted.

That's like saying that a modern calculator and a mechanical arithmometer have very little in common. Sure, the parts are all different, and the construction isn't even remotely similar. They just happen to be doing the same thing.

They are doing "the same thing" only from the point of view of function, which only makes sense from the point of view of the thing utilizing this function (e.g. a clerical worker that needs to add numbers quickly). Otherwise, if "the parts are all different, and the construction isn't even remotely similar", how can the thing they're doing be "the same"? More importantly, how is it possible to make useful inferences…

The more you try to look into the LLM internals, the more similarities you find. Humanlike concepts, language-invariant circuits, abstract thinking, world models.

Mechanistic interpretability is struggling, of course. But what it found in the last 5 years is still enough to dispel a lot of the "LLMs are merely X" and "LLMs can't Y" myths - if you are up to date on the relevant research.

It's not just the outputs. The process is somewhat similar too. LLMs and humans both implement abstract thinking of some kind - much like calculators and arithmometers both implement addition.

Re: Ilya Sutskever: We're moving from the age of scaling to the age of research

#260

Post by myrmidon »

joelthelion wrote: post #227

> When do you expect that impact? I think the models seem smarter than their economic impact would imply. > Yeah. This is one of the very confusing things about the models right now. As someone who's been integrating "AI" and algorithms into people's workflows for twenty years, the answer is actually simple. It takes time to figure out how exactly to use these tools, and integrate them into existing tooling and workf…

Another limitation that I see right now is that for "economic impact" you want the things to have initiative and some agency, and there is well-justified hesitancy in providing that even where possible.

Having a bunch of smart developers that are not allowed to do anything on their own and have to be prompted for every single action is not too advantageous if everyone is human, either ;)

Post reply on HN