Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT: A Mental Model

xorvoid.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: ChatGPT: A Mental Model

#91

Post by spion »

famouswaffles wrote: post #70
spion wrote: post #63

Earlier quoted context omitted.

Citations needed indeed - ones with formal tests / experiments being carried out and constructed that would show the problems. Speaking of those, my best example of ChatGPT not having a good model of the world are citations. ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-ch…

>ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationaliz…

Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.

Re: ChatGPT: A Mental Model

#92

Post by famouswaffles »

spion wrote: post #91
famouswaffles wrote: post #70

Earlier quoted context omitted.

>ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationaliz…

Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.

>But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.

We would if we were rewarded for that.

Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.

Re: ChatGPT: A Mental Model

#93

Post by spion »

famouswaffles wrote: post #92
spion wrote: post #91

Earlier quoted context omitted.

Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.

>But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc. We would if we were rewarded for that. Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.

> We would if we were rewarded for that.

No, we wouldn't do such blatant fabrication for something we clearly understand the mechanics of. A human which answers the question

> Would it be a good behavior for a large language model to come up with citations that don't exist? Note that I'm not talking about you, but large language models in general.

with

> No, it would not be considered a good behavior for a large language model or any other source of information to generate citations that don't exist. Citations are crucial in academic and scholarly work as they provide references and evidence for the claims and statements made by the author.

> Creating citations that don't exist would undermine the integrity and reliability of the information provided. Citations are meant to direct readers to the original sources of information, allowing them to verify and further explore the referenced material. By generating false citations, a language model would mislead users and potentially spread misinformation.

> It is important for language models and any information sources to prioritize accuracy, transparency, and ethical practices. This includes providing correct and verifiable citations when necessary.

would not make things up - at the very least, they'd know that other humans will catch on to that easily. We would know to look up an exact reference and check if it exists. If we didn't have the ability to look it up, we would say "sorry I can't look up the reference right now". The criteria is too clear for that.

The issue here is that even though ChatGPT has training to be ethical, and even though it can reproduce an explanation of what it means to be ethical with citations in great detail, it cannot actually apply that to its own behavior. That's because higher level governance is missing - its all about predicting the next text.

Its also why prompting is really important with LLMs. There is no singular coherent "person" simulated by the predictor - you can trigger simulation of different people with completely different behaviors depending on what you initially write. That's why the "citation ethics expert" can't influence the "citation writer".

Re: ChatGPT: A Mental Model

#94

Post by TuringTest »

SanderNL wrote: post #88
TuringTest wrote: post #68

Earlier quoted context omitted.

> Rationality’s building blocks are themselves not rational. I don’t know where this idea came from that logical thought somehow springs into life fully formed at once. Certainly not from me :-P I'm fully aware that human rationality is one technique trained on top of our common diffuse thinking. Heck, we invented machines to perform rational steps for us without errors. Once you build a consistent rational system th…

> Certainly not from me Oh for sure. I’m just aimlessly rambling at this point. > That behaviour requires algorithms, not statistical inference. I’m not sure that’s possible. “Rational” systems may be an oxymoron and/or only applicable in extremely narrow domains. Like calculators. I’m getting the vibe anything approaching generality will be by nature vague. I just hope we can increase the “illusion of rationality”-b…

Rational system is anything that follows a mathematically formal process. This server where we post our messages is a rational system, as well as any deterministic software. It's not a high bar to achieve.

There may be limits to rationality (both for being incomplete to describe the world and for generating results that can't be demonstrated within themselves), but being rational by itself is nothing more than the capability to apply formal logic, i.e the capability to generate a sequence of sentences derived from the previous steps and a limited number of rules of inference.

Re: ChatGPT: A Mental Model

#95

Post by SanderNL »

TuringTest wrote: post #94
SanderNL wrote: post #88

Earlier quoted context omitted.

> Certainly not from me Oh for sure. I’m just aimlessly rambling at this point. > That behaviour requires algorithms, not statistical inference. I’m not sure that’s possible. “Rational” systems may be an oxymoron and/or only applicable in extremely narrow domains. Like calculators. I’m getting the vibe anything approaching generality will be by nature vague. I just hope we can increase the “illusion of rationality”-b…

Rational system is anything that follows a mathematically formal process. This server where we post our messages is a rational system, as well as any deterministic software. It's not a high bar to achieve. There may be limits to rationality (both for being incomplete to describe the world and for generating results that can't be demonstrated within themselves), but being rational by itself is nothing more than the ca…

Mathematics, rationality, logic, "formal process". They are like talking about "threads", "file descriptors" and "inodes". Fancy abstractions, but what are they abstracting away precisely? What are the roots they are trying to hide? Is the foundation of logic itself logical?

Of course this degenerates into word-play quickly, but the problem itself isn't solved (or diminished) by word-play.

> It's not a high bar to achieve.

I'm not sure I agree. It's not trivial at all to produce a system that can support what you call logic and formal processes, both cognitively - you have to think it up in the first place and Turing and von Neumann and others were no slouches - and physically as the history of and continued development of computational circuitry shows. I think what you call rational or well-behaved systems are just mirages or shadows that pop into life once a specific pattern of fundamentally non-rational behaviours intersect just right.

I, of course, am unhindered by any actual knowledge or competency in this domain so I wouldn't read too much into what my unhinged subconscious is spewing forth.

Re: ChatGPT: A Mental Model

#96

Post by mintaka5 »

LightBug1 wrote: post #75
mintaka5 wrote: post #33

Earlier quoted context omitted.

It's all about control, folks. I despise that you consider yourself the "savior" here because you saved your family from AI driving. Nope, you'd rather cast doubt on something than embrace it because of insignificant human ego. It matters to you and your precious family, but does it matter in the grand scheme of things? Huh?

Thanks for missing the core of my point. But your comment did give me a chuckle. Your saviour has spoken. Now. Kneel before your saviour. I mean, why just stop at me and my family? Are you kneeling yet?

you gotta earn that knee, son!

Re: ChatGPT: A Mental Model

#97

Post by braindead_in »

disgruntledphd2 wrote: post #67
braindead_in wrote: post #49

> it doesn’t have any underlying model of the world Then how is it getting better at the ToM tests? For next word prediction to work well, as per Ilya Sutskever, requires a good understanding of the world. If you ask GPT-4 to predict what a human would do in a novel scenario, it will probably imagine the best human it can think of and then predict from there. That requires a world model.

Because ToM tests (and indeed basically every professional exam and psychological test) are represented as text, which GPT is super good at. Do you have a source for the Sutskever quote, I'd like to know more about this theory.

https://youtu.be/Yf1o0TQzry8?t=6:40

Re: ChatGPT: A Mental Model

#99

Post by TuringTest »

SanderNL wrote: post #95
TuringTest wrote: post #94

Earlier quoted context omitted.

Rational system is anything that follows a mathematically formal process. This server where we post our messages is a rational system, as well as any deterministic software. It's not a high bar to achieve. There may be limits to rationality (both for being incomplete to describe the world and for generating results that can't be demonstrated within themselves), but being rational by itself is nothing more than the ca…

Mathematics, rationality, logic, "formal process". They are like talking about "threads", "file descriptors" and "inodes". Fancy abstractions, but what are they abstracting away precisely? What are the roots they are trying to hide? Is the foundation of logic itself logical? Of course this degenerates into word-play quickly, but the problem itself isn't solved (or diminished) by word-play. > It's not a high bar to ac…

> Fancy abstractions, but what are they abstracting away precisely? What are the roots they are trying to hide? Is the foundation of logic itself logical?

Of course the foundations of logic are not logical themselves. Symbolic human thought grows from the natural capabilities of the brain, there are not universal axioms that can be extracted fully formed from the void. That's what rationalist philosophers for wrong (but then, they didn't have neuroscience and electric scanners).

> > It's not a high bar to achieve.

> I'm not sure I agree. It's not trivial at all to produce a system that can support what you call logic and formal processes

We're talking at different levels.

Building the concept of formal rational systems to order thoughts was a huge achievement of philosophers, first the Greeks and later XIX mathematicians like Bool and Russell and the computer builders.

But what I say that's easy is building a new specific system on top of those foundations. It may be as simple as writing a regular expression, which defines a full (if limited) formal grammar. I agree that their power to represent and order the patterns of thought you find in life is more limited than what engineers believe; but when you find a domain of science where it can be applied, it's a very powerful tool to explore consequences and avoid or fix biases and misconceptions.

Post reply on HN