Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

221–230 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#221

Post by n4r9 »

roenxi wrote: post #198
indigoabstract wrote: post #181

Earlier quoted context omitted.

I am pretty sure that my understanding of something (or lack of) is not encoded into words and probabilities. It's more like a feeling of "I got this figured out" or "I haven't grasped this". Words seem more like a protocol to express some internal model/state in the brain and can never capture the entire actual state, only a small part of it. But since we're not telepaths, we obviously need to use words to exchange…

If brains aren't a complex probability machine, how is it possible that people get the same sort of math problems right and wrong in an inconsistent manner? Or mis-speak? It is undeniable that human reasoning is a stochastic process. Otherwise it wouldn't be reasonable for people to make mistakes after learning something. Especially inconsistent mistakes, like when we give people 10,000 addition problems to do in a r…

All of this seems possible in a deterministic system which evaluates whether or not to retain a piece of information based on past experience.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#222

Post by b1daly »

mo_42 wrote: post #206
zappchance wrote: post #163

Earlier quoted context omitted.

It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?

> If the universe is deterministic, how can punitive justice be justified? Determinism doesn't necessarily mean that organisms always act in the same way. They act in the same way given the exact configuration of them and the world. Obviously, justice changes the configuration of an organism (fines, prison, ...). To me it boils down to the question whether justice decreases the likelihood to commit crimes again. Give…

this is my pet peeve with discussions of ‘free will’ they have an implicit definition—everything being exactly the same at a different time or place—that is non-sensical as far as we know.

I’m still disturbed by peoples confidence in a deterministic universe—I suppose such confidence is based on the success of inductive reasoning but inductive reasoning is a phenomenon based on how our minds work.

As far as I know the philosophical problem of causation is not considered solved?

In any case, elements of randomness seem likely to play a role in human intelligence but what that role is, who knows?

Re: Many in the AI field think the bigger-is-better approach is running out of road

#223

Post by b1daly »

d-cc wrote: post #168
zappchance wrote: post #163

Earlier quoted context omitted.

It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?

The free will "debate", as most consider it, is an utter joke. A tumor or large amount of kinetic energy to the right parts of your, or anybody else's, brain can turn them into an unredeemable monster.

doesn’t your example make clear the existence and nature of free will?

it’s obvious in its absence

Re: Many in the AI field think the bigger-is-better approach is running out of road

#224

Post by NalNezumi »

If the title is correct, a welcoming trend for someone like me trying to apply the advancement in robotics/embodied agent. The bigger the model (generally) the more data required is a constraint very hard to satisfy on a hardware with other requirements (energy consumption, inference speed, memory limitations), not to mention the data collection that is order of magnitude more tedious compares to image and texts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#225

Post by chongli »

MrScruff wrote: post #204
chaosjevil wrote: post #172

Earlier quoted context omitted.

Nah. Human utterances convey purpose on a discursive level; including your comment or mine. We say stuff because we want to do something, like showing [dis]agreement or inform another speaker or change the actions of the other speaker. This is not just probabilistic - it's a way to handle the world. In the meantime those large language models simply predict the next word based on the preceding words.

We're able to do something analogous to reinforcement learning (take on new example data to update our 'weights'). Why do I spend time debating these ideas on Hacker News? Probably the underlying motivation is improving the reliability of my model of the world, which over my lifetime and the lifetimes of creatures before me has led to (somewhat indirectly) positive outcomes in survival and reproduction. Is my model o…

Is my model of the world that different to that of an LLM?

Well, one major way you’re different from an LLM is that you’re alive. You’re capable of learning continuously as you go about your day and interact with the world. LLMs are “dead” in the sense that they’re trained once and frozen, to be used from then on in the exact same state of their initial training.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#226

Post by bemmu »

Tenoke wrote: post #193
indigoabstract wrote: post #181

Earlier quoted context omitted.

I am pretty sure that my understanding of something (or lack of) is not encoded into words and probabilities. It's more like a feeling of "I got this figured out" or "I haven't grasped this". Words seem more like a protocol to express some internal model/state in the brain and can never capture the entire actual state, only a small part of it. But since we're not telepaths, we obviously need to use words to exchange…

The feelings are just how the underlying probabilities are presented to the conscious part of your thinking. You are clearly not consciously noting what your neurons are actually doing.

A feeling is an embedding?

Re: Many in the AI field think the bigger-is-better approach is running out of road

#227

Post by circuit10 »

n4r9 wrote: post #221
roenxi wrote: post #198

Earlier quoted context omitted.

If brains aren't a complex probability machine, how is it possible that people get the same sort of math problems right and wrong in an inconsistent manner? Or mis-speak? It is undeniable that human reasoning is a stochastic process. Otherwise it wouldn't be reasonable for people to make mistakes after learning something. Especially inconsistent mistakes, like when we give people 10,000 addition problems to do in a r…

All of this seems possible in a deterministic system which evaluates whether or not to retain a piece of information based on past experience.

LLMs can be deterministic too at temperature 0

Re: Many in the AI field think the bigger-is-better approach is running out of road

#228

Post by jasmer »

bemmu wrote: post #111
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

[dead]

Re: Many in the AI field think the bigger-is-better approach is running out of road

#229

Post by j-a-a-p »

geysersam wrote: post #124
usrbinbash wrote: post #88

Earlier quoted context omitted.

The problem is, what we call "hallucinating" in LMs isn't a way of creative thinking and coming up with novel solutions. It also has nothing to do with "interpolate and extrapolate". It's simply when the predicted probable sequence isn't grounded in reality. When I ask an LLM to summarize the great water wars of 1999, and how the Trade Union was ultimately defeated by the Antarctic Coalitions hovercraft-fleet under V…

It's a great illusionist. But ultimately it cannot separate relevant information from simple word correlations. > What is heavier, a small floating passenger ferry or a two metric ton heavy rock that sinks to the bottom of the ocean. > A two metric ton heavy rock would be heavier than a small floating passenger ferry. The weight of the rock is two metric tons, which is equivalent to 2,000 kilograms or 4,409 pounds. T…

GPT-4 answer: The weight of an object is determined by its mass, regardless of whether it floats or sinks. So, when you ask which is heavier, a small passenger ferry or a two metric ton heavy rock, it all comes down to the actual mass of the ferry.

A two metric ton rock weighs two metric tons by definition (or 2000 kilograms). However, a small passenger ferry, while it may look small compared to large ferries or ships, can weigh much more than two metric tons. Even a small passenger ferry can weigh dozens or even hundreds of tons, due to the mass of the hull, the engine, and other equipment on board.

So, without specific information about the ferry's mass, it's safe to assume that a "small" passenger ferry is likely heavier than a two metric ton rock. However, if the ferry is particularly small and lightweight, or the term "ferry" is being used to describe a very small watercraft (like a raft or dinghy), it's possible for it to be lighter. You would need the specific weight of the ferry to give a definitive answer.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#230

Post by usrbinbash »

p-e-w wrote: post #155
usrbinbash wrote: post #87

Earlier quoted context omitted.

> and reliably report when they don't know. Then we need a new system, because LMs, no matter if they are large or not, cannot do that, for a very simple reason: A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. And that probability cannot work as a standin for truthfulness, because the LM doesn't produce improbable sequences to begin with...…

> A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. I claim that the human brain doesn't understand "truthfulness" either. It merely creates the impression that understanding is taking place, by adapting to social and environmental pressures. The brain has no "concepts" at all, it just generates output based on its input, its internal wiring,…

> Do you have any evidence that contradicts that claim?

Empirical evidence? Yes I do.

The brain commands an entity that has to exist and function in the context of objective reality. Being unable to verify it's internal state against that, would have been negatively selected some time ago, because stating: "I'm sure that rumbling cave bear with those big sharp teeth is a peaceful herbivore" won't change the objective reality that the caveman is about to become dinner.

How that works in detail is, to the best of my knowledge, still the subject of research in the realm of neurobiology.

Post reply on HN