Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

41–50 of 289 posts

Re: Will scaling work?

#41

Post by whoami_nr »

berniedurfee wrote: post #31

I think there’s a huge assumption here that more LLM will lead to AGI. Nothing I’ve seen or learned about LLMs leads me to believe that LLMs are in fact a pathway to AGI. LLMs trained on more data with more efficient algorithms will make for more interesting tools built with LLMs, but I don’t see this technology as a foundation for AGI. LLMs don’t “reason” in any sense of the word that I understand and I think the ab…

Why next-token prediction is enough for AGI - Ilya Sutskever - https://www.youtube.com/watch?v=YEUclZdj_Sc

Re: Will scaling work?

#42

Post by cortic »

enieslobby wrote: post #38
cortic wrote: post #34

Earlier quoted context omitted.

If humans are basically evolved LLMs, which i think is likely; Reasoning will be an emergent property of LLMs within context with appropriate weights.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

Look at a year old baby, there is no logic, no reasoning, no real consciousness, just basic algorithms and data input ports. It takes ten years of data sets before these emergent properties start to develop, and another ten years before anything of value can be output.

Re: Will scaling work?

#43

Post by lumost »

enieslobby wrote: post #38
cortic wrote: post #34

Earlier quoted context omitted.

If humans are basically evolved LLMs, which i think is likely; Reasoning will be an emergent property of LLMs within context with appropriate weights.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

An LLM is simply a model which given a sequence, predicts the rest of the sequence.

You can accurately describe any AGI or reasoning problem as an open domain sequence modeling problem. It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

Re: Will scaling work?

#44

Post by nnoremap »

gitfan86 wrote: post #13
az226 wrote: post #11

Earlier quoted context omitted.

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

The trick is to make many LLMs work together in feedback loops. Some small some big. That will get us to what was previously known as AGI. The definition of AGI will change, but we will have systems that put perform humans in most ways.

Isaiah 7:14 (NIV): "Therefore the Lord himself will give you a sign: The virgin will conceive and give birth to a son, and will call him Immanuel."

Re: Will scaling work?

#45

Post by lumost »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

The Phi paper and various approaches to distilling from GPT-4 demonstrate that the training data and plausibly order of presentation matter.

The challenge is that we both do not understand which set of data is most beneficial for training, or how it could be efficiently ordered without triggering computationally infeasible problems. However we do know how to massively scale up training.

Re: Will scaling work?

#46

Post by espadrine »

hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

Demis Hassabis of Deepmind echoes a similar sentiment[0]:

> I still think there are missing things with the current systems. […] I regard it a bit like the Industrial Revolution where there was all these amazing new ideas about energy and power and so on, but it was fueled by the fact that there were dead dinosaurs, and coal and oil just lying in the ground. Imagine how much harder the Industrial Revolution would have been without that. We would have had to jump to nuclear or solar somehow in one go. [In AI research,] the equivalent of that oil is just the Internet, this massive human-curated artefact. […] And of course, we can draw on that. And there's just a lot more information there, I think, it turns out than any of us can comprehend, really. […] [T]here's still things I think that are missing. I think we're not good at planning. We need to fix factuality. I also think there's room for memory and episodic memory.

[0]: https://cbmm.mit.edu/video/cbmm10-panel-research-intelligenc...

Re: Will scaling work?

#47

Post by jahnu »

Jabrov wrote: post #32
jahnu wrote: post #19

Earlier quoted context omitted.

> problems that I can absolutely guarantee were not in its training set Can you share the strongest example?

Pretty much any coding problem in a unique or private codebase

I asked for the strongest example the OP can share in order to evaluate their claim. If it's so obvious to the OP that generalisation is happening then it should be easy to provide a strong example, right?

Re: Will scaling work?

#48

Post by tgv »

cortic wrote: post #34
berniedurfee wrote: post #31

I think there’s a huge assumption here that more LLM will lead to AGI. Nothing I’ve seen or learned about LLMs leads me to believe that LLMs are in fact a pathway to AGI. LLMs trained on more data with more efficient algorithms will make for more interesting tools built with LLMs, but I don’t see this technology as a foundation for AGI. LLMs don’t “reason” in any sense of the word that I understand and I think the ab…

If humans are basically evolved LLMs, which i think is likely; Reasoning will be an emergent property of LLMs within context with appropriate weights.

So you think we were originally trained on 300B tokens, those were then ingrained in our synapses, and then we evolved?

Re: Will scaling work?

#49

Post by slibhb »

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the same world with the same general patterns and limitations. With LLMs I expect something similar. Not a singularity, just a new, better tool that, yes, changes things, increases productivity, but leaves human societies more or less the same.

I'd like to be wrong but I can't help but feel that people predicting a revolution are making the same, understandable mistake as my hypothetical 1950s person.

Re: Will scaling work?

#50

Post by berniedurfee »

cortic wrote: post #42
enieslobby wrote: post #38

Earlier quoted context omitted.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

Look at a year old baby, there is no logic, no reasoning, no real consciousness, just basic algorithms and data input ports. It takes ten years of data sets before these emergent properties start to develop, and another ten years before anything of value can be output.

I strongly disagree. Kids, even infants, show a remarkable degree of sophistication in relation to an LLM.

I admit that humans don’t progress much behaviorally, outside of intellect, past our teen years; we’re very instinct driven.

But still, I think even very young children have a spark that’s something far beyond rote token generation.

I think it’s typical human hubris (and clever marketing) to believe that we can invent AGI in less than 100 years when it took nature millions of years to develop.

Until we understand consciousness, we won’t be able to replicate it and we’re a very long way from that leap.

Post reply on HN