Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

11–20 of 289 posts

Re: Will scaling work?

#11

Post by az226 »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a precursor to AGI. They offer a platform to build and scale data sets and test beds.

We haven’t had ML models this large before. There’s innovation in architecture but we often come back to the bitter lesson: more data.

We’re likely going to see experimentation with language models to learn from few examples. Fine tuning pretrained LLMs shows they have quite a remarkable ability to learn from few examples.

Liquid AI has a new learning architecture for dynamic learning and much smaller models.

Some people seem mad about the bitter lesson, they want their model based on human features to work better when so far usually more data wins.

I think the next evolution here is in increasing the quality of the training data and giving it more structure. I suspect the right setup can seed emergent capabilities.

Re: Will scaling work?

#12

Post by dartos »

gchamonlive wrote: post #5
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

I don't think he is wrong. I also don't think the goal of LLMs is to reproduce human intelligence. That is, we don't need human-like inteligence in a box for a tool to be useful. So this assertion could be right and still miss the point of this tech in my opinion. Edit: to expand, if the goal is AGI then yes we need all the help we can get. But even so, AGI is in a totally different league compared to human intellige…

This. I don’t think LLMs are anywhere near sci-fi AGI (think I, robot) It’s such a vague term anyway, AGI.

LLMs provide some really nice text generation, summarization, and outstanding semantic search. It’s drop dead easy to make a natural language interface to anything now.

That’s a big deal. That’s what’s going to give this tech it’s longevity, imo.

Re: Will scaling work?

#13

Post by gitfan86 »

az226 wrote: post #11
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

The trick is to make many LLMs work together in feedback loops. Some small some big.

That will get us to what was previously known as AGI. The definition of AGI will change, but we will have systems that put perform humans in most ways.

Re: Will scaling work?

#14

Post by auggierose »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

Re: Will scaling work?

#15

Post by ralusek »

Almost everything interesting about AI so far has been unexpected emergent behavior, and huge gains through minor insights. While I don't doubt that the current architecture is likely to have a current ceiling below that of peak human intelligence in certain dimensions, it's already surpassed it in some, and there are still gains to be made in others through things like synthetic data.

I also don't understand the claims that it doesn't generalize. I currently use it to solve problems that I can absolutely guarantee were not in its training set, and it generalizes well enough. I also think that one of the easiest ways to get it to generalize better would simply be through giving it synthetic data which demonstrates the process of generalizing.

It also seems foolish to extrapolate on what we have under the assumption that there won't be key insights/changes in architecture as we get to the limitations of synthetic data wins/multi-modal wins.

Re: Will scaling work?

#16

Post by beardedwizard »

az226 wrote: post #11
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

> It’s definitely a precursor to AGI.

What are you basing this claim on? There is no intelligence in an LLM, only humans fooled by randomness.

Re: Will scaling work?

#17

Post by manojlds »

auggierose wrote: post #14
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

We are talking of LLMs, not whether we will be able to reach AGI or not.

Re: Will scaling work?

#18

Post by gchamonlive »

jpk wrote: post #9
gchamonlive wrote: post #5

Earlier quoted context omitted.

I don't think he is wrong. I also don't think the goal of LLMs is to reproduce human intelligence. That is, we don't need human-like inteligence in a box for a tool to be useful. So this assertion could be right and still miss the point of this tech in my opinion. Edit: to expand, if the goal is AGI then yes we need all the help we can get. But even so, AGI is in a totally different league compared to human intellige…

The context of the fine article is scaling LLMs into AGI. It's not about whether the tool is useful or not, as usefulness is a threshold well before AGI. Some folks are spooked that LLMs are a few optimizations away from the singularity, and the article just discusses some reasons why that probably isn't the case.

The article is really good! I was responding to "is he wrong" part of the comment, not the article itself.

Re: Will scaling work?

#19

Post by jahnu »

ralusek wrote: post #15

Almost everything interesting about AI so far has been unexpected emergent behavior, and huge gains through minor insights. While I don't doubt that the current architecture is likely to have a current ceiling below that of peak human intelligence in certain dimensions, it's already surpassed it in some, and there are still gains to be made in others through things like synthetic data. I also don't understand the cla…

> problems that I can absolutely guarantee were not in its training set

Can you share the strongest example?

Re: Will scaling work?

#20

Post by Eddy_Viscosity2 »

auggierose wrote: post #14
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

But we didn't use airplanes to get there. It needed a new approach, different propulsion, different fuel, different attitude control, etc. etc.

LLM may be a necessary step to get to AGI, but it (probably) won't be the one that achieves that goal.

Post reply on HN