Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I disagree with Geoff Hinton regarding "glorified autocomplete"

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

141–150 of 279 posts

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#141

Post by Jensson »

xcv123 wrote: post #136
Jensson wrote: post #132

Earlier quoted context omitted.

> This is more sophisticated than a Markov process. Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes. Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the…

The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. > Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers The attention mechanism…

> The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.

What, yes it is.

> The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.

The context window is finite, so that is the previous step. You know the dumb markov chains that are based on bag of words? They also look several words back, they don't just go based on a single word. LLMs are just that but with a way larger lookback and some extra logic there, but none of that changes the fundamental parts to make it no be a markov process.

With a large enough context size you could argue it is now fundamentally different in practice, but in theory it is the same. There is no "hidden state", its just the previous n words defines the next word.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#142

Post by Jensson »

bytefactory wrote: post #139
Jensson wrote: post #113

Earlier quoted context omitted.

A big difference between a game like Go and writing text is that text is single player. I can write out the entire text, look at it and see where I made mistakes on the whole and edit those. I can't go back in a game of Go and change one of my moves that turned out to be a mistake. So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while al…

This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.

But it can't improve its answer after it has written it, that is a major limitation. When a human writes an article or response or solution, that is likely not the first thing the human thought of, instead they write something down and works on it until it is tight and neat and communicates just what the human wants to communicate.

Such answers will be very hard for an LLM to find, instead you mostly get very verbose messages since that is how our current LLM thinks.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#143

Post by WillPostForFood »

11235813213455 wrote: post #3

I still want to read a ChatGPT text that doesn't look stupidly verbose

Give this GPT called Briefly a try. Responses come back as bullet point lists.I prefer more verbose, but interesting to see this in contrast.

https://chat.openai.com/g/g-LNsEQH5rz-briefly

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#144

Post by bytefactory »

Jensson wrote: post #142
bytefactory wrote: post #139

Earlier quoted context omitted.

This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.

But it can't improve its answer after it has written it, that is a major limitation. When a human writes an article or response or solution, that is likely not the first thing the human thought of, instead they write something down and works on it until it is tight and neat and communicates just what the human wants to communicate. Such answers will be very hard for an LLM to find, instead you mostly get very verbose…

Completely agree. The System 1/System 2 distinction seems relevant here. As powerful as transformers are with just next-token generation and context, which can be hacked to form a sort of short-term memory, some time of real-time learning + long-term memory storage seems like an important research direction.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#145

Post by bytefactory »

Falkon1313 wrote: post #131

I think that insight is an important feature that GPT doesn't seem to have, at least not yet. For instance, I've seen people saying they can ask it for help with how to code something and it will help them. Although the results aren't perfect, they can be helpful. However, I recall years ago asking a more senior developer how to do something. They could've just told me how, but instead they asked why I was doing that…

> I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

I actually think this is a limitation of the RLHF that GPT has been put through. With open-ended questions, I've seen GPT4 come up with reasonable alternatives instead of just answering the question I've asked. This is often seen as the infamous, "however, please consider..." bits that it tacks on, which occasionally do consider actual insights into the problem I'm trying to solve.

In most cases it seems to try very hard to mold the answer into what I want to hear, which in many cases isn't necessarily the best answer. A more powerful version of GPT with a less-restrictive RLHF seems like it would be more open to suggesting novel solutions, although this is just my speculation.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#148

Post by astrange »

lsy wrote: post #126
famouswaffles wrote: post #122

Earlier quoted context omitted.

>The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts. To the bird who can…

That's the difference though. I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. To a neural network there is no distinction. The closest the entire dynamic comes is the very basic annotation of RLHF which itself is done by an external human who is providing the value judgment, but even that is abse…

There is no such thing as a world model, and you don't have one of them. This is a leftover bad psychological concept from the 70s AI researchers who never got anywhere. People and other creatures do very little modeling things, they mostly just do stuff.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#149

Post by xcv123 »

Jensson wrote: post #141
xcv123 wrote: post #136

Earlier quoted context omitted.

The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. > Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers The attention mechanism…

> The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. What, yes it is. > The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property. The c…

If you define the state broadly enough to include the state of the entire machine itself, including all of its internal representations, weights, activations, etc, then you are playing a funny trick here.

By the same reasoning, a human brain is also Markov process.

What you are doing here is a vast oversimplification and it is practically useless for understanding how LLMs work.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#150

Post by tazjin »

lsy wrote: post #116
robbrown451 wrote: post #99

I agree with Hinton, although a lot hinges on your definition of "understand." I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go. Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they f…

The difference is that "the world" is not exhaustible in the same way as Go is. While it's surely true that the number of possible overall Go game states is extremely large, the game itself is trivially representable as a set of legal moves and rules. The "world model" of the Go board is actually just already exhaustive and finite, and the computer's work in playing against itself is to generate more varied data with…

The world on our plane of existence absolutely is exhaustible, just on a much, much larger scale. Doesn't mean that the process is fundamentally different, and for the human perspective there might be diminishing returns.
Post reply on HN