Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

81–90 of 289 posts

Re: Will scaling work?

#81

Post by bee_rider »

eru wrote: post #73
imachine1980_ wrote: post #65

Earlier quoted context omitted.

I feel you are mixing value capture with value generation. If GM produces cars with the same level of margins as Facebook or Google, things will be different. LVMH (Louis Vuitton Group) holds a value equivalent to that of Toyota, Volkswagen, and two-thirds of Ford combined. Louis Vuitton alone was valued more than Red Hat a few months ago. This doesn't mean that Louis Vuitton is more valuable than Red Hat, but rather…

> This doesn't mean that Louis Vuitton is more valuable than Red Hat, but rather that it captures Value more effectively than Red Hat. What definition of 'valuable' are you using here?

Probably something like market cap (although I guess it would have to be based on the past now that Red Hat has been bought), or there are nebulous measures of brand value out there.

I think it is a fair point TBH, my original comment could have been more clear about this aspect.

Re: Will scaling work?

#82

Post by collaborative »

nemo44x wrote: post #25

Where in the hype cycle are we for LLMs? Are we in the late stages of the rise or over the peak and beginning the slide?

LLMs are still too expensive to run and therefore can't be supported by ads. If costs get lower we'll see them being pushed _a lot_ more

Re: Will scaling work?

#83

Post by nmca »

YetAnotherNick wrote: post #8

> ‘5 OOMs off’ I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of people don't have any content on public web except for personal photos. A minority has few public social media posts and it is rare for people to write blog or research paper etc. And almost everyone has some content written in mail or docs or messaging.

From the article, and relevant here:

I’m worried that when people hear ‘5 OOMs off’, how they register it is, “Oh we have 5x less data than we need - we just need a couple of 2x improvements in data efficiency, and we’re golden”. After all, what’s a couple OOMs between friends?

No, 5 OOMs off means we have 100,000x less data than we need.

Re: Will scaling work?

#84

Post by bee_rider »

jsnell wrote: post #28

The original title ("will scaling work?") seems like a much more accurate description of the article than the editorialized "why scaling will not work" that this got submitted with. The conclusion of the article is not that scaling won't work! It's the opposite, the author thinks that AGI before 2040 is more likely than not.

It might be nice to modify the title a bit though, to indicate that it is about AGI.

Obviously scaling works in general, just ask anyone in HPC, haha.

Re: Will scaling work?

#85

Post by timacles »

cortic wrote: post #42
enieslobby wrote: post #38

Earlier quoted context omitted.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

Look at a year old baby, there is no logic, no reasoning, no real consciousness, just basic algorithms and data input ports. It takes ten years of data sets before these emergent properties start to develop, and another ten years before anything of value can be output.

This comment is just sad. What are you even talking about? Have you ever seen a 1 year old

Re: Will scaling work?

#86

Post by visarga »

Der_Einzige wrote: post #33
nsagent wrote: post #22

Earlier quoted context omitted.

I mentioned this to another commenter as well: You might want to reconsider your stance on emergent abilities in LLMs considering the NeurIPS 2023 best paper winner is titled: "Are Emergent Abilities of Large Language Models a Mirage?" https://arxiv.org/abs/2304.15004 https://blog.neurips.cc/2023/12/11/announcing-the-neurips-20...

Papers which get accepted with honors are not necessarily more truthful than papers which have been rejected. Yann LeCunn goes on twitter like any other grad student around NeurIPS or ICML/ICMR and bitterly complains when one of his (many) papers is rejected. Whose more likely to be correct here? Yann LeCunn (the TOP nlp scholar in our field by citations, who does claim that most emergent capabilities are real in oth…

Gebru and her "Stochastic Parrots" did a big disservice to AI safety turning the debate into a shit-show of identity politics. Now she has her own institute, it was a move up for her career. Her twitter spats with Yann LeCun were legendary. Literally sent him to educate himself and refused to debate him.

Re: Will scaling work?

#87

Post by david_allison »

lysecret wrote: post #77
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

Good point to me the internet was just "other people", what differentiated is not the 4 people you know but literally (almost) and potentially all other people. With AI, the way I see it, it is just virtual other people. Of course, a bit stranger but more simillar than you think.

There's currently little to no learning or feedback loop due to the relatively small context window sizes.

I've done many language exchanges with people using Google Translate and the lack of improvement/memory of past conversations is a real motivation killer; I'm concerned this will move on to general discourse on the internet with the proliferation of LLMs.

I'm sure many people have already gone around in circles with rules-based customer support. AI can make this worse.

Re: Will scaling work?

#88

Post by revskill »

No, human is not that intelligent to generate super intelligent bot in a short time.

My estimation is about 200 years in future to have a "human-brain AI" that works.

All idea should be treated equally, not based on revenue metrics. If everyone could make a Youtube clone, the revenue should be divided equally to all of creator, that's the way the world should move forward, instead of monopoly.

Everything will be suck, forever.

Re: Will scaling work?

#89

Post by timacles »

lumost wrote: post #43
enieslobby wrote: post #38

Earlier quoted context omitted.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

An LLM is simply a model which given a sequence, predicts the rest of the sequence. You can accurately describe any AGI or reasoning problem as an open domain sequence modeling problem. It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

> It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

The real world is random, requires making decisions on incomplete information in situations that have never happened before. The real world is not a sequence of tokens.

Consciousness requires instincts in order to prioritize the endless streams of information. One thing people dont want to accept about any AI is that humans always have to tell it WHAT to think about. Our base reptilian brains are the core driver behind all behavior. AI cannot learn that

Re: Will scaling work?

#90

Post by herval »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

> And yet, 70 years later, things have certainly changed, but we're living in the same world with the same general patterns and limitations. With LLMs I expect something similar. Not a singularity, just a new, better tool that, yes, changes things, increases productivity, but leaves human societies more or less the same.

by what criteria do you see the world as the same today vs 70 years ago?

Post reply on HN