Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

21–30 of 289 posts

Re: Will scaling work?

#21

Post by nsagent »

az226 wrote: post #11
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

You might want to reconsider your stance on emergent abilities in LLMs considering the NeurIPS 2023 best paper winner is titled:

"Are Emergent Abilities of Large Language Models a Mirage?"

https://arxiv.org/abs/2304.15004 https://blog.neurips.cc/2023/12/11/announcing-the-neurips-20...

Re: Will scaling work?

#22

Post by nsagent »

ralusek wrote: post #15

Almost everything interesting about AI so far has been unexpected emergent behavior, and huge gains through minor insights. While I don't doubt that the current architecture is likely to have a current ceiling below that of peak human intelligence in certain dimensions, it's already surpassed it in some, and there are still gains to be made in others through things like synthetic data. I also don't understand the cla…

I mentioned this to another commenter as well:

You might want to reconsider your stance on emergent abilities in LLMs considering the NeurIPS 2023 best paper winner is titled:

"Are Emergent Abilities of Large Language Models a Mirage?"

https://arxiv.org/abs/2304.15004 https://blog.neurips.cc/2023/12/11/announcing-the-neurips-20...

Re: Will scaling work?

#23

Post by MPSimmons »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

I don't think the data is the weakness.

We're using Transformer architecture right now. There's no reason there won't be further discoveries in AI that are as impactful as "Attention is All You Need".

We may be due for another "AI Winter" where we don't see dramatic improvement across the board. We may not. Regardless, LLMs using the Transformer architecture may not have human level intelligence, but they _are_ useful, and they'll continue to be useful. In the 90s, even during the AI winter, we were able to use Bayesian classification for such common tasks as email filtering. There's no reason we can't continue to use Transformer architecture LLMs for common purposes too. Content production alone makes it worth while.

We don't _need_ AGI, it just seems like the direction we are heading as a species. If we don't get there, it's fine. No need to throw the baby out with the bath water.

Re: Will scaling work?

#24

Post by ImHereToVote »

manojlds wrote: post #17
auggierose wrote: post #14

Earlier quoted context omitted.

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

We are talking of LLMs, not whether we will be able to reach AGI or not.

Airplanes in this analogy are essentially the collection of matrix multiplications that emulate reasoning in a very rough but useful manner in an LLM.

It's unclear whether a rocket ship is a multimodal neural net. Or some sort of swarm of LLM's in an adversarial relationship, or something completely novel. Regardless, we might be as far between LLM's to ASI's, as airplanes are to rocket ships. Or not.

Re: Will scaling work?

#26

Post by crowbahr »

ralusek wrote: post #15

Almost everything interesting about AI so far has been unexpected emergent behavior, and huge gains through minor insights. While I don't doubt that the current architecture is likely to have a current ceiling below that of peak human intelligence in certain dimensions, it's already surpassed it in some, and there are still gains to be made in others through things like synthetic data. I also don't understand the cla…

Latest research shows emergent behavior is illusory. It doesn't preclude future emergence but currently models show 0 emergent behavior.

To me the most interesting aspect of LLMs is the way that they reveal cognitive 0-days in humans.

The human race needs patches to cognitive firmware to deal with predictive text... Which is a fascinating revelation to me. Sure it's backed up by psych analysis for decades but it's interesting to watch it play out on such a large scale.

Re: Will scaling work?

#27

Post by viraptor »

gitfan86 wrote: post #6
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

Over the past year there have been advances in making models smaller while keeping performance high. So if that continues then he is wrong unless he is defining LLMs in a strict way that does not include new improvement in the future

For an example, the diagrams in the post compare the big gpts, but looking at the number of tokens PHI-2 sits below gpt3. And it still beats it in Humaneval and a few other benchmarks.

Re: Will scaling work?

#28

Post by jsnell »

The original title ("will scaling work?") seems like a much more accurate description of the article than the editorialized "why scaling will not work" that this got submitted with. The conclusion of the article is not that scaling won't work! It's the opposite, the author thinks that AGI before 2040 is more likely than not.

Re: Will scaling work?

#29

Post by PlasmonOwl »

Author is leveraging mental inflexibility to generate an emotional response of denial. Sure, his points are correct but are constrained. Let’s remove 2 constraints and reevaluate:

1 - Babies learn much more with much less 2 - Video training data can be made in theory at incredible rates

The questions becomes: why is the author focusing on approaches in AI investigated in like 2012? Does the author think SOTA is text only? Are OpenAI or other market leaders only focusing on text? Probably not.

Re: Will scaling work?

#30

Post by Der_Einzige »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

Even the largest LLM has had less "total information" than most humans take in through all of their senses over their lifetime. A single day for a baby is taking in a continuous stream of among other things high quality video and audio and does a large amount of processing on that. Much of that for very young babies is unsupervised learning (clustering), where baby learns that object A and object B are different despite knowing nothing else about their properties.

Humans can learn using every ML learning paradigm in ever modality: unsupervised, self-supervised, semi-supervised, supervised, active, reinforcement based, and anything else I might be missing. Current LLMs are stuck with "self-supervised" with the occasional reinforced (RLHF) or supervised (DPO) cherry on top at the end. non multi-modal LLMs operate with one modality. We are hardly scratching the surface on what's possible with multi-modal LLMs today. We are hardly scratching the surface for training data for these models.

The overwhelming majority of todays LLMs are vastly undertrained and exhibit behavior of undertrained systems.

The claim from the OP about scale not giving us further emergent properties flies in the face of all of what we know about this field. Expect further significant gains despite nay-sayers claiming it's impossible.

Post reply on HN