Will scaling work?
dwarkeshpatel.com
Will scaling work?
1–10 of 289 posts
Re: Will scaling work?
#2Why wouldn't you include the "LLM" part in the title?
Hint for everyone else here:
It's about scaling LLMs.
Re: Will scaling work?
#3Why wouldn't you include the "LLM" part in the title? Hint for everyone else here: It's about scaling LLMs.
After reading the article, I really enjoyed it and the believer + skeptic perspective. However, I only touched it because I thought "meh, what is there going to be about web scaling".
Re: Will scaling work?
#4>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can add to an airplane to make it reach the moon.
Never thought about it in this sense. Is he wrong?
Re: Will scaling work?
#5>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…
I don't think he is wrong. I also don't think the goal of LLMs is to reproduce human intelligence. That is, we don't need human-like inteligence in a box for a tool to be useful. So this assertion could be right and still miss the point of this tech in my opinion.
Edit: to expand, if the goal is AGI then yes we need all the help we can get. But even so, AGI is in a totally different league compared to human intelligence, they might as well be a different species.
Re: Will scaling work?
#6>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…
Over the past year there have been advances in making models smaller while keeping performance high.
So if that continues then he is wrong unless he is defining LLMs in a strict way that does not include new improvement in the future
Re: Will scaling work?
#7>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…
he's not wrong, and yet he's not right.
Re: Will scaling work?
#8> ‘5 OOMs off’
I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of people don't have any content on public web except for personal photos. A minority has few public social media posts and it is rare for people to write blog or research paper etc. And almost everyone has some content written in mail or docs or messaging.
Re: Will scaling work?
#9>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…
I don't think he is wrong. I also don't think the goal of LLMs is to reproduce human intelligence. That is, we don't need human-like inteligence in a box for a tool to be useful. So this assertion could be right and still miss the point of this tech in my opinion. Edit: to expand, if the goal is AGI then yes we need all the help we can get. But even so, AGI is in a totally different league compared to human intellige…
The context of the fine article is scaling LLMs into AGI. It's not about whether the tool is useful or not, as usefulness is a threshold well before AGI. Some folks are spooked that LLMs are a few optimizations away from the singularity, and the article just discusses some reasons why that probably isn't the case.
Re: Will scaling work?
#10>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…
I don't think he is wrong. I also don't think the goal of LLMs is to reproduce human intelligence. That is, we don't need human-like inteligence in a box for a tool to be useful. So this assertion could be right and still miss the point of this tech in my opinion. Edit: to expand, if the goal is AGI then yes we need all the help we can get. But even so, AGI is in a totally different league compared to human intellige…
We don’t need human-like intelligence in a box for a tool to be useful,
But human-like intelligence is what many companies are spending billions to try and achieve