Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

251–260 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#251

Post by endisneigh »

p-e-w wrote: post #155
usrbinbash wrote: post #87

Earlier quoted context omitted.

> and reliably report when they don't know. Then we need a new system, because LMs, no matter if they are large or not, cannot do that, for a very simple reason: A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. And that probability cannot work as a standin for truthfulness, because the LM doesn't produce improbable sequences to begin with...…

> A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. I claim that the human brain doesn't understand "truthfulness" either. It merely creates the impression that understanding is taking place, by adapting to social and environmental pressures. The brain has no "concepts" at all, it just generates output based on its input, its internal wiring,…

I find these takes so lazy. What you have claimed here is just totally wrong.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#252

Post by tempodox »

thomasahle wrote: post #185
tempodox wrote: post #159

Earlier quoted context omitted.

Only you, a human developer can be truly creative. An LLM can only ever reproduce what it has seen before.

> An LLM can only ever reproduce what it has seen before. Anyone who's played around with these models know that at least some generalization is taking place.

So generalization counts as creativity in your book?

Re: Many in the AI field think the bigger-is-better approach is running out of road

#253

Post by imtringued »

JimtheCoder wrote: post #30
stavros wrote: post #29

Earlier quoted context omitted.

Search has definitely been dead since before LLMs, we just don't have a replacement yet.

I mean more the concept of search, not the current implementation

Search is about finding existing results. The opposite of search is to start from first principles and work your way up until you have created the desired result yourself.

In the real world, search reduces information acquisition costs as you only have to spend time and resources on finding an existing result rather than recreating it.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#254

Post by elif »

After a year of Tesla FSD beta I agree 100%. I used to think that adding more and more and more real world video events to the training pool was contributing at least a diminishing value to the model.

But the worst of all behaviors have not diminished and some have actually gotten worse. Most of the improvements now come through what feels like manual heuristics and tuning parameters rather than any actual improvement in intelligence.

I sincerely hope there is a path forward that involves meaningfully culling data which is producing bad behavior, stronger guardrails, and/or a new paradigm in how the model is built from the data entirely, as I don't see a path to level 3 by simply growing the existing model.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#255

Post by TeMPOraL »

imtringued wrote: post #243
TeMPOraL wrote: post #42

Earlier quoted context omitted.

How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and…

And that is exactly why the current generation of AI is the same as all the other generations. It is a bruteforce attempt to solve something that shouldn't need brute force.

I used to believe that, but I'm no longer convinced. At least when you're trying to approximate the way humans think, and especially human language, there may not be a simpler way - our brains are themselves a product of randomness, and are unlikely to factor nicely into theoretically clean components. It might be computationally cheaper to repeat the process than trying to figure it out and encode in analytical form.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#256

Post by imtringued »

4ndrewl wrote: post #51
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

They don't know that they don't know. It's only hallucination from a human's perspective. From the model's perspective it's _all_ hallucination.

That is true. There is no difference between fiction and nonfiction for the LLM.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#257

Post by ruszki »

fho wrote: post #235
roenxi wrote: post #198

Earlier quoted context omitted.

If brains aren't a complex probability machine, how is it possible that people get the same sort of math problems right and wrong in an inconsistent manner? Or mis-speak? It is undeniable that human reasoning is a stochastic process. Otherwise it wouldn't be reasonable for people to make mistakes after learning something. Especially inconsistent mistakes, like when we give people 10,000 addition problems to do in a r…

> It is undeniable that human reasoning is a stochastic process It can still be a deterministic process. If anything came out of the whole LLM story for me it is that I am even more convinced that it is. My (somewhat educated, but still naive) idea why it looks like a stochastic process is that the brain gets incredible amounts of random input. We literally get bombarded with particles and energy ever instance we liv…

According to physics, it must be stochastic. The only question is distribution, and whether we can ignore outlier cases.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#258

Post by imtringued »

skepticATX wrote: post #34
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

This is ultimately just very powerful semantic search though, is it not? It seems that what we need to make a big leap forward is better reasoning. There is a lot of debate between the GPT-4 can/can't reason camps, but I haven't seen anyone try to argue that it reasons particularly well.

It shouldn't even be that difficult to build it. Modus ponens and good theorems are all you need.

Building the data set for that should be quite trivial.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#259

Post by p-e-w »

usrbinbash wrote: post #230
p-e-w wrote: post #155

Earlier quoted context omitted.

> A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. I claim that the human brain doesn't understand "truthfulness" either. It merely creates the impression that understanding is taking place, by adapting to social and environmental pressures. The brain has no "concepts" at all, it just generates output based on its input, its internal wiring,…

> Do you have any evidence that contradicts that claim? Empirical evidence? Yes I do. The brain commands an entity that has to exist and function in the context of objective reality. Being unable to verify it's internal state against that, would have been negatively selected some time ago, because stating: "I'm sure that rumbling cave bear with those big sharp teeth is a peaceful herbivore" won't change the objective…

That's speculation, not evidence. The traits you describe aren't demonstrably incompatible with the mechanism I proposed.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#260

Post by tkgally »

Naive question, perhaps: What roles are multimodal models likely to play in the future of AI?

The comments here and in the Economist article seem to be about only large language models. In the initial announcement of GPT-4 in March, OpenAI described it as “a large multimodal model (accepting image and text inputs, emitting text outputs),” but they haven’t yet released the image part to the public.

What will happen when models are trained not only on text, and not only on text and images, but also on video, audio, chemical analyses of air and other substances in our surroundings, tactile data from devices that explore the physical world, etc.?

Post reply on HN