Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

231–240 of 289 posts

Re: Will scaling work?

#231

Post by Xelynega »

PlasmonOwl wrote: post #29

Author is leveraging mental inflexibility to generate an emotional response of denial. Sure, his points are correct but are constrained. Let’s remove 2 constraints and reevaluate: 1 - Babies learn much more with much less 2 - Video training data can be made in theory at incredible rates The questions becomes: why is the author focusing on approaches in AI investigated in like 2012? Does the author think SOTA is text…

Isn't 1 a point for their "skeptic" persona?

If babies learn much more from much less, isn't that evidence that the LLM approach isn't as efficient as whatever approach humans implement biologically, so it's likely LLM processes won't "scale to ago"?

For video data, that's not how LLMs work(or any NNs for that matter). You have to train them on what you want them to look at, so if you want them to predict the next token of text given an input array, you need to train it on the input arrays and output tokens.

You can extract the data in the form you need from the video content, but presumably that's already been done for the most part, since video transcripts are likely included in the training data for gpt.

Re: Will scaling work?

#232

Post by Xelynega »

lern_too_spel wrote: post #174
bob1029 wrote: post #170

I think the "self-play" path is where the scary-powerful AI solutions will emerge. This implies persistence of state and logic that lives external to the LLM. The language model is just one tool. AGI/ASI/whatever will be a system of tools, of which the LLM might be the least complicated one to worry about. In my view, domain modeling, managing state, knowing when to transition between states, techniques for final dec…

It's not necessary for the author's purpose of providing more data. We're only training on one kind of input so far, text, from which these models have built some understanding of the world. Humans train on more inputs, and the data to provide those inputs for training a model is readily available, in far larger quantities than individual human brains consume. Data is not the issue.

We're training on text because that's what we're making the model do.

It's a fact of neural networks that to train them supervised you need the training data in the expected input for(vector of n thousand preceding tokens for LLMs) with the expected output(the next token for LLMs). "Training them on video" would mean converting the video to a format we can train the llm with, then training the LLM with that info.

This would probably be a 1 OOM increase at maximum, if the video transcripts aren't already a part of the training data for gpt.

Re: Will scaling work?

#233

Post by Xelynega »

gitfan86 wrote: post #164
discreteevent wrote: post #146

Earlier quoted context omitted.

Parent is saying that with something as sophisticated as intelligence it's not enough to say that if it behaves like a duck it's a duck (which is what your seem to be saying and which the parent calls a 0-day). There are some really good bulshitters who have led smart people into deep trouble. These bulshitters behaved really like ducks but they weren't ducks. The duck test just isn't good enough. The -1 day is where…

That is a new definition of intelligence that you are using. You are saying that even when something can outperform humans in the SAT or other tests of intelligence, it isn't actually intelligent due to it not being a carbon based lifeform

Outperforming on the SAT(a test that the model was trained on) doesn't seem like a marker of intelligence as much as it is of recall.

Re: Will scaling work?

#234

Post by Xelynega »

Jabrov wrote: post #32
jahnu wrote: post #19

Earlier quoted context omitted.

> problems that I can absolutely guarantee were not in its training set Can you share the strongest example?

Pretty much any coding problem in a unique or private codebase

When I talk to people that use copilot to improve their coding workflow, what I often hear is that copilot can replace boilerplate but not any business logic specific to the problem being solved.

It sounds like you have a different experience(copilot is useful for business logic). Do you have maybe any examples of what you mean by "any coding problems"?

Is it similar to what I've heard previously(copilot can replace boilerplate) or have LLMs actually solved business problems for you in code?

Re: Will scaling work?

#235

Post by qudat »

lnxg33k1 wrote: post #183
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

Imagine telling those same people in the 50s that all those changes in productivity would come for the benefit of no one since the work week would be the same and purchasing power would decline

Such a wild take. Would you want to live in the 50s? I definitely would not.

Re: Will scaling work?

#236

Post by qudat »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

What do you think would need to be different for it to be considered meaningful to you?

Re: Will scaling work?

#237

Post by beardedwizard »

az226 wrote: post #209
beardedwizard wrote: post #16

Earlier quoted context omitted.

> It’s definitely a precursor to AGI. What are you basing this claim on? There is no intelligence in an LLM, only humans fooled by randomness.

Precursor as in it will help in synthetic data generation, testing, etc. at scale that gives us more powerful models. It is a necessary intermediate step on our path to AGI.

Fair enough, it could be but I'm not sure it has to be in any way...

Re: Will scaling work?

#238

Post by defrost »

qudat wrote: post #235
lnxg33k1 wrote: post #183

Earlier quoted context omitted.

Imagine telling those same people in the 50s that all those changes in productivity would come for the benefit of no one since the work week would be the same and purchasing power would decline

Such a wild take. Would you want to live in the 50s? I definitely would not.

Some of us didn't have a choice.

I just got back from the shop with my dad, he was born in 1935 .. neither of us struggled to survive the 1950s.

It's dropped from a standard 40 hour work week to a 38 hour week for indexing a living wage, but things are more or less still as they were in 1907 (inflation adjusted) albeit with greater choice of consumerables.

https://www.fwc.gov.au/about-us/history/waltzing-matilda-and...

Re: Will scaling work?

#239

Post by esafak »

Jensson wrote: post #192
cortic wrote: post #184

Earlier quoted context omitted.

You know i assumed that was true until right now. But I can't think of a single example of reason and intelligence existing without any form of language. Even insects have rudimentary language, and in fact reasoning and intelligence seem to scale with the complexity of language, both by species and within species.

Do slime mold have a language? Slime mold can learn and adapt to environments, so it is intelligent and can do rudimentary reasoning, but I doubt it communicates that information to other slime molds. It is a very different kind of life form though so many things that applies to other complex being doesn't apply to them. Being a large single cell means that they learn by changing its proteins and other internals, ver…

I would say language is necessarily discrete, or digital. Slime molds communicate in analog.

Re: Will scaling work?

#240

Post by esafak »

nnoremap wrote: post #44
gitfan86 wrote: post #13

Earlier quoted context omitted.

The trick is to make many LLMs work together in feedback loops. Some small some big. That will get us to what was previously known as AGI. The definition of AGI will change, but we will have systems that put perform humans in most ways.

Isaiah 7:14 (NIV): "Therefore the Lord himself will give you a sign: The virgin will conceive and give birth to a son, and will call him Immanuel."

"The Shoe Is The Sign!"

-- Monty Python.

Post reply on HN