Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I disagree with Geoff Hinton regarding "glorified autocomplete"

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

131–140 of 279 posts

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#131

Post by Falkon1313 »

I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

For instance, I've seen people saying they can ask it for help with how to code something and it will help them. Although the results aren't perfect, they can be helpful.

However, I recall years ago asking a more senior developer how to do something. They could've just told me how, but instead they asked why I was doing that. Then they told me not to bother with writing code at all, and to instead do this other, much simpler thing which would have the same effect at solving the client's request. ChatGPT wouldn't have had that insight.

Years later, a junior dev came to me asking for help with his code because it wasn't working. We went over the code line by line and he explained what it was trying to do and it all looked good. But when he ran it, the web server crashed. I told him to try adding a comment:

    // Do not remove this comment.
Sure enough, it worked perfectly and the server no longer crashed. Why? Because I realized that if his code was wrong he should get an error message, not crash the server. But sometime back I had read about a bug in a version of the interpreter that would crash when parsing a file that was an exact multiple of 4096 bytes. Would chatGPT have thought of that? Probably not. It would've just talked about the code.

Which is not to say that it's useless. But it lacks greater context and the insight to go beyond the parameters of the question on its own initiative. Then again, so do humans most of the time.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#132

Post by Jensson »

xcv123 wrote: post #130
mbrumlow wrote: post #5

> If you want to be really good, you have to understand what’s being said. That’s the only way. This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or req…

By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

> This is more sophisticated than a Markov process.

Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes.

Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#133

Post by famouswaffles »

lsy wrote: post #126
famouswaffles wrote: post #122

Earlier quoted context omitted.

>The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts. To the bird who can…

That's the difference though. I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. To a neural network there is no distinction. The closest the entire dynamic comes is the very basic annotation of RLHF which itself is done by an external human who is providing the value judgment, but even that is abse…

>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins.

Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception.

And yes LLMs can in fact tell truth from fiction.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

Your argument seems to boil down to "they can't perform experiments" but that isn't true either.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#134

Post by mannykannot »

albertzeyer wrote: post #10
mbrumlow wrote: post #5

> If you want to be really good, you have to understand what’s being said. That’s the only way. This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or req…

> You would realize that all we are doing is using statistics to predict what the next item might be. So what does "understanding" really means then? "Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding…

I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.

For example, we are aware that some, but by no means all, of what people say is about an external world that may or may not conform to what the words say. We can also doubt that we have understood things correctly, and take steps to either confirm or refute our opinions. We see ourselves as entities in an external reality containing other individuals who also do this, and that we, and they, have a limited ability to influence what happens in that world. Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#135

Post by civilized »

IshKebab wrote: post #108
civilized wrote: post #52

Earlier quoted context omitted.

It is very easy to separate humans from LLMs. Humans created math without being given all the answers beforehand. LLMs can't do that yet. When an LLM can create math to solve a problem, we will be much closer to AGI.

Some humans created maths. And it took thousands of years of thinking and interaction with the real world. Seems like goalpost moving to me. I think the real things that separate LLMs from humans at the moment are: * Humans can do online learning. They have long term memory. I guess you could equate evolution to the training phase of AI but it still seems like they don't have quite the same on-line learning capabilit…

About this goalpost moving thing. It's become very popular to say this, but I have no idea what it's supposed to mean. It's like a metaphor with no underlying reality.

Did a wise arbiter of truth set up goalposts that I moved? I guess I didn't get the memo.

If the implied claim is "GPT would invent math too given enough time", go ahead and make that claim.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#136

Post by xcv123 »

Jensson wrote: post #132
xcv123 wrote: post #130

Earlier quoted context omitted.

By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

> This is more sophisticated than a Markov process. Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes. Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the…

The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.

> Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers

The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#137

Post by TerrifiedMouse »

brookst wrote: post #97
TerrifiedMouse wrote: post #93

Earlier quoted context omitted.

I don’t think LLMs work towards anything. It just picks from a list of likely “next tokens” at random.

“Likely” as defined by…?

Statistics as defined by training data.

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#138

Post by bytefactory »

SkiFire13 wrote: post #121
robbrown451 wrote: post #99

I agree with Hinton, although a lot hinges on your definition of "understand." I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go. Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they f…

> One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (o…

> It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move

Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?

I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#139

Post by bytefactory »

Jensson wrote: post #113
robbrown451 wrote: post #99

I agree with Hinton, although a lot hinges on your definition of "understand." I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go. Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they f…

A big difference between a game like Go and writing text is that text is single player. I can write out the entire text, look at it and see where I made mistakes on the whole and edit those. I can't go back in a game of Go and change one of my moves that turned out to be a mistake. So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while al…

This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.

Re: I disagree with Geoff Hinton regarding "glorified autocomplete"

#140

Post by hackinthebochs »

mannykannot wrote: post #134
albertzeyer wrote: post #10

Earlier quoted context omitted.

> You would realize that all we are doing is using statistics to predict what the next item might be. So what does "understanding" really means then? "Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding…

I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton. For example, we are aware that some, but by no means all, of what people say is about an external world that may or may not conform to what the words say. We can also doubt that we have understood things correctly, and take steps t…

>I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.

Agreed. What LLMs say about understanding deserves a lot more attention than it has received. I wrote down some of my thoughts on the matter:

https://www.reddit.com/r/naturalism/comments/1236vzf

>Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?

These two points aren't necessarily in opposition, and understanding why is I think key to solving a lot of important problems around intelligence, sentience, etc. To compute is to operate on formal properties. But this doesn't exclude semantic properties from having causal relevance to the behavior of the system. What we need is a way to conceptualize how a system can have multiple related descriptions at different levels. A description at the level of semantics doesn't exclude a description in terms of formal properties or vice versa. I think of it in terms of constraints: the higher level descriptions constrain the lower level behavior. What the computational description does is ensures the higher level semantic constraint is maintained. It does this by the particular space of computational dynamics it follows. Essentially, the information that picks out this programs space of branching dynamics embeds the semantic description in question, and this description realizes the computational dynamic necessary to maintain the higher level semantic constraint. Rather than semantics being in opposition to formal properties, they are two sides of the same coin.

Post reply on HN