Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT as a Calculator for Words

simonwillison.net

Post reply on HN
Print view View original

61–70 of 81 posts

Re: ChatGPT as a Calculator for Words

#61

Post by teekert »

"The ChatGPT model is huge, but it’s not huge enough to retain every exact fact it’s encountered in its training set."

That's because there is no way for the model to take the internet and separate fact from fiction or truth from falsehood. So it should not even try to, unless it can somehow weigh options (or preform its own experiments). And that doesn't mean counting occurrences, it means figuring out a coherent worldview and using it as a prior to interpret information, and then still acknowledging that it could be wrong.

Re: ChatGPT as a Calculator for Words

#62

Post by Al0neStar »

tbalsam wrote: post #11
skybrian wrote: post #9

Earlier quoted context omitted.

I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball. Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.

I find this to be a better explanation than "it's just regurgitating strings of text" No, it is clearly not, and that is a very easily testable hypothesis. Thank you for sharing.

testable how?

Re: ChatGPT as a Calculator for Words

#63

Post by skybrian »

ftxbro wrote: post #20
skybrian wrote: post #9

Earlier quoted context omitted.

I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball. Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.

The essay is long and complicated so I'm not sure how much of it you read closely, but it specifically addresses this distinction between the simulator and the simulacrum. In the analogy of the essay, your argument would be like saying that reality cannot be simply the application of quantum physics, because you are allowed to make new rules like Calvinball within reality which are different from the rules of quantum…

I do understand the difference between a simulator and what's being simulated. I still think they got it all wrong, that the simulator is better called a "writer," the simulated world is better called a "story," and the agent is a better called a "fictional character."

We know there's no deeper level to the simulation/game because we have the entire "game history" (the chat history) and we understand it in approximately same way that the LLM does. (That's what the LLM was trained to do, understand and respond to text the same way we do.) We know that the bot has no hidden state when it's not the bot's turn because of how the bot's API works.

So there's nowhere for a deeper simulation to live. It's as shallow as it looks.

More:

https://skybrian.substack.com/p/ai-chats-are-turn-based-game...

Re: ChatGPT as a Calculator for Words

#64

Post by tbalsam »

Al0neStar wrote: post #62
tbalsam wrote: post #11

Earlier quoted context omitted.

I find this to be a better explanation than "it's just regurgitating strings of text" No, it is clearly not, and that is a very easily testable hypothesis. Thank you for sharing.

testable how?

out of distribution tests. if the concept holds consistently over a long period, the concept is the stable thing. If not, then it's only memorizing string densities.

For a number of years we've been basically showing the first to be the case, especially as the model is scaled and the context increases, differentially against the second. String density probabilities can be surprisingly brittle, to be honest. The curse of dimensionality applies to them too, believe it or not, which I believe is why topic discussion, reasoning, and integration over longer distances of text is that differential test that shows pretty clearly that substring memorization/text density stuff is not 'just' what the model is learning. Because mathematically/statistically/from an information density perspective/etc etc otherwise it would be basically impossible, I think.

That's my best understanding, at least.

Re: ChatGPT as a Calculator for Words

#66

Post by blueblimp »

skybrian wrote: post #9
ftxbro wrote: post #2

As a long time LLM enjoyer I just want to mention https://generative.ink/posts/simulators/ as I think it's by far the most insightful take on the GPT LLMs even though it was from before ChatGPT. It's better than blurry jpeg and stochastic parrot etc.

I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball. Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.

I've always thought of them a bit like improv, since they tend to follow the "yes, and..." rule, by happily continuing whatever direction you want to go. Now that the base models have been fine-tuned to avoid some topics, that's less true than it used to be, but it still feels like the most natural mode of operation.

Re: ChatGPT as a Calculator for Words

#68

Post by jazzyjackson »

hbarka wrote: post #56

Confabulate is a better word than hallucinate. How did “hallucinating” get popularized? It’s a terrible term in this context.

I feel like it had something to do with DeepDream - in the popular consciousness, tripping on acid / hallucinating became something that computers are surprisingly good at, and maybe that transferred to text models.

Re: ChatGPT as a Calculator for Words

#70

Post by jazzyjackson »

dror wrote: post #53
anotherpaulg wrote: post #45

Earlier quoted context omitted.

Gish looks really nice. I'm going to give it a try. It seems like you've been using similar workflows to what I've been trying for coding with gpt? https://github.com/paul-gauthier/easy-chat#created-by-chatgp...

Pretty much, except that I'm automating everything as much as I can, so that I just give the instructions and GPT does the rest. Here's an example: ----- #import ~/work/gish/tasks/coding.txt Change the following so that it looks for the open AI key in the following fashion: 1. env variable 2. os.home()/.openai 3. Throws an exception telling the user to put it in one of the above, and then exits #diff ~/work/gish/src/…

fabulous
Post reply on HN