Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

501–510 of 518 posts

Re: What is ChatGPT doing and why does it work?

#501

Post by jmoak3 »

jmoak3 wrote: post #87
codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

I saw a great comment here, and I will repeat it without the attribution it deserves: We may have realized it's easier to build a brain than to understand one

Attribution to jonnycomputer:

https://news.ycombinator.com/item?id=34008075

Re: What is ChatGPT doing and why does it work?

#502

Post by the_af »

williamcotton wrote: post #498
the_af wrote: post #494

Earlier quoted context omitted.

> No, but it is correctly running the best move functions so through induction we can see it will successfully play a full game. I'm not convinced induction applies. ChatGPT tends to "go astray" in conversations where it needs to maintain state; even with your patch for this (essentially reminding it what the state is at every prompt) I would test it just to make sure it can run a game through completion, make good m…

> I'm not convinced induction applies. ChatGPT tends to "go astray" in conversations where it needs to maintain state; even with your patch for this (essentially reminding it what the state is at every prompt) I would test it just to make sure it can run a game through completion, make good moves all the way, and be able to tell when the game is over. You don't seem to understand what I am saying. ChatGPT cannot main…

We seem to be talking at cross purposes. I understand (at a very high level) what LLMs do, and I don't think they can do actual computation.

Why do you insist on things I've already said I understand? I know ChatGPT is not good at maintaining state -- though it can fake it convincingly (which understandably, seems to trip people up). I think it looks at your chat history within the session in order to generate the next response, which is why it can "degenerate" within a single session (but also, it's how it can fake and make it seem it's keeping state, by looking at the whole history before each reply).

I don't understand the rest of your answer. You seem to be really upset at "the people".

PS:

> So by induction once we know that the bestMove function is correct

"By induction", nope. Prove it. Run an actual full game instead of arguing with me. It will take you shorter to play the game than to debate with me.

Re: What is ChatGPT doing and why does it work?

#503

Post by the_af »

williamcotton wrote: post #499
the_af wrote: post #494

Earlier quoted context omitted.

> No, but it is correctly running the best move functions so through induction we can see it will successfully play a full game. I'm not convinced induction applies. ChatGPT tends to "go astray" in conversations where it needs to maintain state; even with your patch for this (essentially reminding it what the state is at every prompt) I would test it just to make sure it can run a game through completion, make good m…

Like, we could both be thinking and talking about things like, “I wonder which programming languages are better or worse for these tasks? Is it harder to translate to NASM or ARM64? Or C? Or Lisp? Which lisp performs better? What’s the relationship between training data and programming languages and is this separate from an inherent complexity of a programming language? Can we use LLMs to make objective measurements…

> But no, instead you want to prove to me that ChatGPT is some parlor trick…

Excuse me, what?

I'm sorry, I've zero interest in discussing NASM or Lisp or whatnot. This was about the limitations of ChatGPT, not whatever strikes your fancy.

Re: What is ChatGPT doing and why does it work?

#504

Post by theGnuMe »

IanCal wrote: post #383
wruza wrote: post #320

Earlier quoted context omitted.

Research papers turned into a mouth tape isn’t cool either. The commenter above noted that it’s not specifically a game model that emerged in an LLM in that paper, but 64 additional networks that somehow represent each tile from values of all weights/activations of an LLM. Which is somewhat weaker than the initial claim, slightly less obvious. People nowadays use papers as a means to shut someone up with a summary, b…

> The commenter above noted that it’s not specifically a game model that emerged in an LLM in that paper, but 64 additional networks that somehow represent each tile from values of all weights/activations of an LLM. Which is somewhat weaker than the initial claim, slightly less obvious. I don't follow. Those are used to identify the game model. They test that they've found an internal model by then altering the state…

Can a game state be encoded as a set of weights? Sure. These can be in the MLP part after the LLM. Sure. You could train this directly with fine-tuning and then have text prompts to move things. That is pretty standard I think and that would be supervised learning. In terms of unsupervised learning, I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) and what the next likely token is just from training it as an LLM. After all tokens are just placeholders and the relationships between them are encoded in the text. That patterns of activation are akin to remembering the 8k tokens in the prompt or what not.

Re: What is ChatGPT doing and why does it work?

#505

Post by theGnuMe »

wruza wrote: post #339
netsharc wrote: post #169

Earlier quoted context omitted.

I asked Google Home what the definition of self-awareness is, and it says "conscious knowledge of one's character s and feelings.". But me saying "ChatGPT surely doesn't have feelings, so it can't be self-aware!" would be a simple cop-out/gotcha response. I guess it's a Chinese Room, that when you ask about Chinese Rooms, can tell you what those things are. I almost said the word "aware" there, but the person in the…

Therapy sometimes uses a method called exposition. E.g. if one has an irrational fear of elevators, they can gradually expose themselves to it. Stand before it then leave. Call it and look inside. Enter it on the first floor and exit without riding. After few weeks or months they can start using it, because the fear response reduces to manageable levels. Because nothing bad happens (feedback). One may condition thems…

I think you meant "exposure" therapy rather than exposition.

Re: What is ChatGPT doing and why does it work?

#506

Post by kazinator »

the_af wrote: post #502
williamcotton wrote: post #498

Earlier quoted context omitted.

> I'm not convinced induction applies. ChatGPT tends to "go astray" in conversations where it needs to maintain state; even with your patch for this (essentially reminding it what the state is at every prompt) I would test it just to make sure it can run a game through completion, make good moves all the way, and be able to tell when the game is over. You don't seem to understand what I am saying. ChatGPT cannot main…

We seem to be talking at cross purposes. I understand (at a very high level) what LLMs do, and I don't think they can do actual computation. Why do you insist on things I've already said I understand? I know ChatGPT is not good at maintaining state -- though it can fake it convincingly (which understandably, seems to trip people up). I think it looks at your chat history within the session in order to generate the ne…

What's the difference between keeping state and looking at the chat history?

Keeping state is something a human would have to do, because for a human, it would be very tedious and slow to re-read the history to recover context, relative to the timeliness expectation of the interlocutor.

Re: What is ChatGPT doing and why does it work?

#507

Post by IanCal »

theGnuMe wrote: post #504
IanCal wrote: post #383

Earlier quoted context omitted.

> The commenter above noted that it’s not specifically a game model that emerged in an LLM in that paper, but 64 additional networks that somehow represent each tile from values of all weights/activations of an LLM. Which is somewhat weaker than the initial claim, slightly less obvious. I don't follow. Those are used to identify the game model. They test that they've found an internal model by then altering the state…

Can a game state be encoded as a set of weights? Sure. These can be in the MLP part after the LLM. Sure. You could train this directly with fine-tuning and then have text prompts to move things. That is pretty standard I think and that would be supervised learning. In terms of unsupervised learning, I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) an…

> Can a game state be encoded as a set of weights?

It's not in the weights because the weights don't change.

> These can be in the MLP part after the LLM. Sure

I'm not even sure what this means. The mlps are not used at all by the network.

> I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) and what the next likely token is just from training it as an LLM.

You might not but the idea that they are just outputting based on sequences without having an internal model of the world is a common one. This experiment was a test to get more information on that question.

> After all tokens are just placeholders and the relationships between them are encoded in the text.

They don't tell you the state of the board.

Re: What is ChatGPT doing and why does it work?

#508

Post by theGnuMe »

IanCal wrote: post #507
theGnuMe wrote: post #504

Earlier quoted context omitted.

Can a game state be encoded as a set of weights? Sure. These can be in the MLP part after the LLM. Sure. You could train this directly with fine-tuning and then have text prompts to move things. That is pretty standard I think and that would be supervised learning. In terms of unsupervised learning, I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) an…

> Can a game state be encoded as a set of weights? It's not in the weights because the weights don't change. > These can be in the MLP part after the LLM. Sure I'm not even sure what this means. The mlps are not used at all by the network. > I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) and what the next likely token is just from training it as an…

Sorry by weights, I really meant the pattern of activations... I should have made that more clear. But the weights are trained by the game transcripts to produce activation patterns that could represent the board state. Or it could be local position patterns learnt during training. Positional representation (attention) of the N-1 tokens in the autoregressive task. Did they look at the attention patterns? Anyway there is a recent PhD from Stanford who looked at CNNs with SAT similarly and presented some evidence that the activations patterns can be decoded to determine the satisfying solution.

Re: What is ChatGPT doing and why does it work?

#509

Post by vivegi »

criddell wrote: post #356
vivegi wrote: post #224

Earlier quoted context omitted.

The print version has the Recent Writings rendered at the end of the document. On my machine with "US Letter" page size and "No header/footer" I got 72 pages. The main article was 69 pages and the last three were the "cruft" (which includes the Recent Writings). A quick hack might be to postprocess the PDF (eg: using Ghostscript) and trim the last three pages, if all you want is the main article.

Well that’s strange. If I print to PDF I get the article exactly as it’s laid out on screen. I did get what I want with the OneNote clipper. Edit: Now when I try to print this article to PDF from my phone or iPad, the browser crashes immediately. Something weird is going on with my devices...

It is probably something to do with Safari/WebKit (assuming you are printing from your iPhone/iPad). The page uses a media query for print in the CSS sheet linked to it. That does most of the heavy lifting for laying out the print page.

Re: What is ChatGPT doing and why does it work?

#510

Post by IanCal »

theGnuMe wrote: post #508
IanCal wrote: post #507

Earlier quoted context omitted.

> Can a game state be encoded as a set of weights? It's not in the weights because the weights don't change. > These can be in the MLP part after the LLM. Sure I'm not even sure what this means. The mlps are not used at all by the network. > I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) and what the next likely token is just from training it as an…

Sorry by weights, I really meant the pattern of activations... I should have made that more clear. But the weights are trained by the game transcripts to produce activation patterns that could represent the board state. Or it could be local position patterns learnt during training. Positional representation (attention) of the N-1 tokens in the autoregressive task. Did they look at the attention patterns? Anyway there…

> . But the weights are trained by the game transcripts to produce activation patterns that could represent the board state

A slight phrasing thing here just to be clear - the model is not trained to produce a representation of the board state explicitly. It is never given [moves] = [board state] and it is not trained on correctly predicting the board state by passing it in like [state] + move. The only thing that is trained on that is the probes, which is done after the training of OthelloGPT and does not impact what the model does.

Their argument is that the state is represented in the activation patterns and that this is then used to determine the next move, are you countering that to suggest it instead may be "local position patterns learnt during training. Positional representation (attention) of the N-1 tokens in the autoregressive task"?

If the pattern of activations did not correspond to the current board state, modifying those activations to produce a different internal model of the board wouldn't work. I also don't follow how the activations would mirror the expected board state.

Post reply on HN