Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

101–110 of 289 posts

Re: Will scaling work?

#101

Post by measured_step »

timacles wrote: post #89
lumost wrote: post #43

Earlier quoted context omitted.

An LLM is simply a model which given a sequence, predicts the rest of the sequence. You can accurately describe any AGI or reasoning problem as an open domain sequence modeling problem. It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

> It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem. The real world is random, requires making decisions on incomplete information in situations that have never happened before. The real world is not a sequence of tokens. Consciousness requires instincts in order to prioritize the endless streams of information. One thing people dont want to accept about any AI is t…

How do our base reptilian brains reason? We don't know the specifics, but unless it's magic, then it's determined by some kind of logic. I doubt that logic is so unique that it can't eventually be reproduced in computers.

Re: Will scaling work?

#102

Post by HarHarVeryFunny »

HarHarVeryFunny wrote: post #92
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

I think AGI can change the world once it gets way beyond human level both in terms of types of beyond-human "senses" and pattern matching/prediction (i.e. intelligence), but we are nowhere near that yet. On their current trajectory LLMs are just expert systems that will let certain types of simple job be automated. A potential productivity amplifier similar to having a personal assistant that you can assign tasks too…

Odd to see this down-voted... I guess my prediction of the future has rubbed someone the wrong way, but if you disagree then why not just reply ?!

Re: Will scaling work?

#103

Post by tw1984 »

LLM is going to bring tons of cool applications, but AGI is not an application!

You can feed your dog 100,000 times a day, but that won't make it a 1,000kg dog. The whole idea that AGI can be achieved by predicting the next word is just pure marketing nonsense at best.

Re: Will scaling work?

#104

Post by lumost »

RandomLensman wrote: post #53
lumost wrote: post #43

Earlier quoted context omitted.

An LLM is simply a model which given a sequence, predicts the rest of the sequence. You can accurately describe any AGI or reasoning problem as an open domain sequence modeling problem. It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

In the broader sense that is tricky as accurate prediction is not always the right metric (otherwise we'd still be using epicycles for the planets).

It depends on the goal, epicycles don't tell you about the nature of heavenly bodies - but they do let you keep an accurate calendar for a reasonable definition of accurate. I'm not sure whether I need deep understanding of intelligence to gain economic benefit from AI.

Re: Will scaling work?

#105

Post by toast0 »

HarHarVeryFunny wrote: post #96
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I would say that it feels different because the internet / smartphones are more about giving everyone access to inexpensive, high bandwidth, communication (nearly) everywhere. But high bandwidth communications have been available everywhere for a long time, if you had a need and were willing to pay for it --- tv news would bounce signals off a satelite for on scene reports, etc.

Re: Will scaling work?

#106

Post by elktown »

visarga wrote: post #69
beepbooptheory wrote: post #57

Earlier quoted context omitted.

While this is certainly true, I think we can't ignore the intense enthusiasm and faith of a large cohort of our peers (or, you know, HN commenters) who believe this to be The Way, and are not necessarily stakeholders in any meaningful sense. Just look at some of the responses even in this thread. It feels like some people just need this, and respond to balanced skepticism as Alyosha does to his brother Ivan. In part,…

There is no magic in the brain. There is no magic in LLMs. There is just new experience we gain by interacting with the environment and society. And there is the trove of past experience encoded in our books. We got smart by collecting experience, in other words, from outside. The magic in the brain was not in the brain, but everywhere else. What is experience? We are in state S, and take action A, and observe feedba…

> There is no magic in the brain.

The amount of hubris we have in our field is deeply embarrassing. Imagine a neuroscientists reading that. The thought makes me blush.

Re: Will scaling work?

#107

Post by YetAnotherNick »

nmca wrote: post #83
YetAnotherNick wrote: post #8

> ‘5 OOMs off’ I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of people don't have any content on public web except for personal photos. A minority has few public social media posts and it is rare for people to write blog or research paper etc. And almost everyone has some content written in mail or docs or messaging.

From the article, and relevant here: I’m worried that when people hear ‘5 OOMs off’, how they register it is, “Oh we have 5x less data than we need - we just need a couple of 2x improvements in data efficiency, and we’re golden”. After all, what’s a couple OOMs between friends? No, 5 OOMs off means we have 100,000x less data than we need.

I meant 100,000x. At least for everyone I know, they have 100,000x data in mail/messaging/docs/notes/meeting etc. than their blog or any public site they own. Hell I would even say that if you just have all the meetings of zoom, it will be few order of magnitude higher than the entire public web.

Re: Will scaling work?

#108

Post by bee_rider »

HarHarVeryFunny wrote: post #96
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I don’t know really, I was a kid in the 90’s.

This is a bit far from the economic aspect, but the world currently seemed to be utterly suffused with a looming sense of dread, I think because we have, or know other people have, news notifications in their pockets telling us all about how bad things are.

I don’t remember that feeling from the 90’s, but then, I was a kid. And of course before that there was the constant fear of nuclear annihilation, which we’ve only recently brought back really. Maybe growing up in the end of history warped my perspective, haha.

Re: Will scaling work?

#109

Post by syndacks »

auggierose wrote: post #14
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

Sorry, but what’s with HN’s obsession with analogies? You see this in almost every comment section where someone tries to dis/prove a point using an analogy. I get the allure but it’s intellectually brittle; by the time someone starts to argue off the second or third incantation of the original analogy, the forest has been lost for the tree.

Re: Will scaling work?

#110

Post by Aerbil313 »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

Technology is the one force that drives modern human societies, Western ones even more. The world has changed dramatically, especially with smartphones. I suggest reading Ted Kaczynski.
Post reply on HN