Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

61–70 of 289 posts

Re: Will scaling work?

#61

Post by _a_a_a_ »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

> but we're living in the same world with the same general patterns and limitations

seems odd. What 'patterns' and 'limitations' do you still see? Because I see so much has changed.

Re: Will scaling work?

#62

Post by blackoil »

beardedwizard wrote: post #16
az226 wrote: post #11

Earlier quoted context omitted.

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

> It’s definitely a precursor to AGI. What are you basing this claim on? There is no intelligence in an LLM, only humans fooled by randomness.

> only humans fooled by randomness

Is there another kind?

Re: Will scaling work?

#63

Post by bamboozled »

whoami_nr wrote: post #41
berniedurfee wrote: post #31

I think there’s a huge assumption here that more LLM will lead to AGI. Nothing I’ve seen or learned about LLMs leads me to believe that LLMs are in fact a pathway to AGI. LLMs trained on more data with more efficient algorithms will make for more interesting tools built with LLMs, but I don’t see this technology as a foundation for AGI. LLMs don’t “reason” in any sense of the word that I understand and I think the ab…

Why next-token prediction is enough for AGI - Ilya Sutskever - https://www.youtube.com/watch?v=YEUclZdj_Sc

Ilya can feel the AGI

Re: Will scaling work?

#64

Post by andyg_blog »

Really stellar, well-sourced article that comes across as unbiased as possible. I especially enjoyed the almost-throw-away link to "the bitter lesson" near the end, the gist of which is: "Methods that leverage massive compute to capture intrinsic complexity always outperform humans' attempts to encode that complexity by hand"

Re: Will scaling work?

#65

Post by imachine1980_ »

bee_rider wrote: post #60
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

I feel you are mixing value capture with value generation. If GM produces cars with the same level of margins as Facebook or Google, things will be different. LVMH (Louis Vuitton Group) holds a value equivalent to that of Toyota, Volkswagen, and two-thirds of Ford combined. Louis Vuitton alone was valued more than Red Hat a few months ago. This doesn't mean that Louis Vuitton is more valuable than Red Hat, but rather that it captures Value more effectively than Red Hat.

Re: Will scaling work?

#66

Post by visarga »

spzb wrote: post #56
cortic wrote: post #42

Earlier quoted context omitted.

Look at a year old baby, there is no logic, no reasoning, no real consciousness, just basic algorithms and data input ports. It takes ten years of data sets before these emergent properties start to develop, and another ten years before anything of value can be output.

Have you ever met a baby? They're nothing like an LLM. For starters, they learn without using language. By one year old they've taught themselves to move around the physical world. They've started to learn cause and effect. They've learned where "they" end and "the rest of the world" begins. All an LLM has "learnt" is that some words are more likely to follow others.

Why not? We have multi-modal models as well. Not pure text.

Re: Will scaling work?

#67

Post by crowbahr »

gitfan86 wrote: post #39
crowbahr wrote: post #26

Earlier quoted context omitted.

Latest research shows emergent behavior is illusory. It doesn't preclude future emergence but currently models show 0 emergent behavior. To me the most interesting aspect of LLMs is the way that they reveal cognitive 0-days in humans. The human race needs patches to cognitive firmware to deal with predictive text... Which is a fascinating revelation to me. Sure it's backed up by psych analysis for decades but it's in…

When a human makes a mistake it is a "cognitive 0-day" but when an LLM does something correctly it is "illusory"?

The cognitive 0-day is not the way that humans act like LLMs, it's the way humans anthropomorphize LLMs. It's the blind faith that LLMs do more than they do.

The illusion of emergence is fact not fiction. The cognitive biases exposed by stochastic parrots are fact not fiction.

Re: Will scaling work?

#69

Post by visarga »

beepbooptheory wrote: post #57
mgaunard wrote: post #37

I think the more interesting question is how long will people cling to the illusion that LLMs will lead us to AGI? Maintaining the illusion is important to keep the money flowing in.

While this is certainly true, I think we can't ignore the intense enthusiasm and faith of a large cohort of our peers (or, you know, HN commenters) who believe this to be The Way, and are not necessarily stakeholders in any meaningful sense. Just look at some of the responses even in this thread. It feels like some people just need this, and respond to balanced skepticism as Alyosha does to his brother Ivan. In part,…

There is no magic in the brain. There is no magic in LLMs. There is just new experience we gain by interacting with the environment and society. And there is the trove of past experience encoded in our books. We got smart by collecting experience, in other words, from outside. The magic in the brain was not in the brain, but everywhere else.

What is experience? We are in state S, and take action A, and observe feedback R. The environment is the teacher, giving us reward signals. We can only increase our knowledge incrementally, by trying our many bad ideas, and sometimes paying with our lives. But we still leave morsels of newly acquired experience for future generations.

We are experience machines, both individually and socially. And intelligence is the distilled experience of the past, encoded in concepts, methods and knowledge. Intelligence is a collective process. None of us could reach our current level without language and society.

Human language is in a way smarter than humans.

Post reply on HN