Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

461–470 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#461

Post by SanderNL »

toss1 wrote: post #455
SanderNL wrote: post #442

Earlier quoted context omitted.

May I ask why? How is this not “paternalistic”? Why protect the people, from what? Is this worse than what they are already exposed to on a daily basis? Does facebook say it fucks you up and makes you an addict? Why this urge to enforce “truth in advertising”? I am not saying you shouldn’t do it, it’s just that I do not get why and the article doesn’t mention it. It is somehow a given that The People need to be prote…

>>How is this not “paternalistic”? I see three categories of handling this issue, perhaps call them "Paternalistic", "Responsible", and "Anti-Social". - Paternalistic would be like: "We assess [thing] to be dangerous in a number of ways and so we forbid you to use [thing], except by using our high priesthood representatives as intermediaries". - Responsible would be like: We assess [thing] to be dangerous in a number…

Interesting. Thank you for the detailed response. Paternalistic is indeed not the right term and it’s nice to worry about your fellow humans to this degree. I guess I am anti-social.

But I disagree on the premise that this is comparable to a dangerous boobytrap. It’s not completely without danger of course, but that’s a property it shares with kitchen utensils, cars, scissors, religious texts..

Are we going to preface the Bible with this kind of warnings too? I’m all for it, but be consistent.

I also think that warnings like that are cheap ways to evade liability and do not actually solve the issue which is people being stupid.

Your example is not about people being deliberate and thinking deeply about their actions based on given information. If people kill themselves after talking to a chatbot I do not think a textual warning would have sufficed.

> it is that there are potential and serious harms that are easily avoided if they are told

Let’s agree to disagree.

Sorry for sounding obtuse. I enjoyed your input, makes me think. I’m just a classic annoying neckbeard.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#462

Post by TuringTest »

sltkr wrote: post #140
chaxor wrote: post #84

Earlier quoted context omitted.

Can you prove that it actually "doesn't know anything"? What do you mean by that? Being critical does not make you educated on the subject. There are so many comments like this, yet never provide any useful information. Saying there's no value in something, as everyone seems to try to do regarding LLMs, should come with more novel insights than parroting this same idea along with every single person on HN.

Especially ironic that the original comment came from a user named "TuringTest"!

Yup, I've been thinking about this topic since long before large language model existed :P

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#463

Post by TuringTest »

chaxor wrote: post #84
TuringTest wrote: post #66

Large language models have read everything, and they don't know anything. They are excellent imitators, being able to clone the style and contents of any subject or source you ask for. When you prompt them, they will uncritically generate a text that combines the relevant topics in creative ways, without the least understanding of their meaning. Their original training causes them to memorize lots of concepts , both…

Can you prove that it actually "doesn't know anything"? What do you mean by that? Being critical does not make you educated on the subject. There are so many comments like this, yet never provide any useful information. Saying there's no value in something, as everyone seems to try to do regarding LLMs, should come with more novel insights than parroting this same idea along with every single person on HN.

I have not said there is no value in LLMs, quite the contrary.

What I'm warning is against thinking of them as independent agents with their own minds, because they don't work like that at all, so you'd be anthropomorphising them.

These models certainly have a compilation of knowledge, but it is statistical knowledge - in the same way as a book of logarithms has lots of mathematical knowledge, but you wouldn't say that the book 'knows logarithms'. The compilation contains statistical 'truths' about the topics on which it has been trained; and contrary to a written book, that knowledge can be used operationally to build new information.

Yet that static knowledge does not reach the point of having a will of its own; there is nothing in the content generation system that makes it take decisions or establish its own objectives from its statistical tables of compiled knowledge.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#464

Post by TuringTest »

chaxor wrote: post #188
j0057 wrote: post #149

Earlier quoted context omitted.

That's easy: ask it anything, then "correct" it with some outrageous nonsense. It will apologize (as if to express regret), and say you're correct, and now the conversation is poisoned with whatever nonsense you fed it. All form and zero substance. We fall for it because normally the use of language is an expression of something, with ChatGPT language is just that, language, with no meaning. To me that proves knowing…

This isn't always true, especially not with gpt-4. Also, this isn't a proof really, or even evidence to show that it doesn't 'know' something. It appears to reason well about many tasks - specifically "under the hood" (reasoning not explicitly stated within the output provided). Yes, of course "it's just a language model" is spouted over and over and is sometimes true (though obviously not for gpt-4), but that statem…

'Reasoning' (as in deriving new statements with precision, following logical inference rules) is precisely what the large language models can't do.

It's better to think of this models as 'generating' chains of relevant words, where 'relevant' is defined by similarity of those areas of knowledge on which it has been trained, and which are "activated" as close to the topics in the prompt. Which is not at all dissimilar to how humans learn about a new topic, btw.

This way, by "activating" concepts of areas of knowledge and finding words that are more likely than others to fit those concepts, the model is able to create texts following the constraints you instruct it with - such as poems that rhyme, or critical analysis of scientific articles.

The most important point to be aware of is that this creation model is completely different to how automatic reasoning models create content, which is by having a formal representation of a knowledge domain and creating logical inferences that can be mathematically proven correct within the model. A reasoning model cannot lie, but it cannot create content beyond the logical implications of its premises; its quite the opposite of what language models do.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#465

Post by cultureswitch »

japhyr wrote: post #57
thewataccount wrote: post #51

Earlier quoted context omitted.

What do you use it for? I'm assuming code related? I've found it useful for some boilerplate + writing tests and making some script and some documentation. I'm curious what you or others that use it all day use it for especially if it's not for programming?

I was just working on a small exploratory project in Python. I used sys.argv because it's so quick to prototype with. When I started refining the project for longer term development, I wanted to convert the CLI to use argparse, so I could build a more nuanced CLI. I gave GPT a couple example commands I wanted to use, and in less than a minute I had a fully converted CLI that did exactly what I wanted, with more consi…

See, a better LLM or an experienced human would have told you to use docopt instead of argparse and saved you more time in the long run.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#466

Post by f6v »

dwaltrip wrote: post #445
f6v wrote: post #426

Earlier quoted context omitted.

> dumb algorithms I’m not an expert and haven’t seen architecture of ChatGPT. But it’s probably far from dumb. You can’t simply put more data into any algorithm and expect improvement. You need a model to be able to take advantage of the data and train in reasonable time. That’s exactly what’s been happening in ML field.

A huge portion of the improvements in each successive version of gpt came from increasing the model size and the amount of training data. The research trends indicate that there still are additional improvements to be found by continuing to scale things up, especially when it comes to the amount of training data. Not indefinitely, obviously. But we haven’t exhausted the available data sources yet, and the curves for…

> Do you have specific reasons for thinking that gpt-4 is near the end of this trend?

I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data.

I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image models include many tricks, like inception layers.

I suspect there’re similar challenges in LLMs. And I’m not sure whether current ChatGPT can use more data to become better at reasoning, or at some point they’d need to come up with different architectures.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#467

Post by whimsicalism »

f6v wrote: post #466
dwaltrip wrote: post #445

Earlier quoted context omitted.

A huge portion of the improvements in each successive version of gpt came from increasing the model size and the amount of training data. The research trends indicate that there still are additional improvements to be found by continuing to scale things up, especially when it comes to the amount of training data. Not indefinitely, obviously. But we haven’t exhausted the available data sources yet, and the curves for…

> Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data. I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image…

You are largely wrong, to be honest.

Mostly, you can simply stack more transformers and (as long as you have layer norm), it can train on more data (of fixed sequence length).

There are some tricks & tips, but the vast majority lie outside of the architecture improvements.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#468

Post by whimsicalism »

f6v wrote: post #394
dwaltrip wrote: post #312

Earlier quoted context omitted.

Gpt-4 hallucinates meaningfully less than gpt-3. There is more evidence in favor of “more improvements to come” than “ai winter approaches”. A lot smart people seem to be saying that the existing approaches have room to improve simply by training with more data. Based on what I’ve read and roughly speculating, it looks there is easily enough existing data for gpt-5 and probably a few more versions after. I’m not sure…

> A lot smart people seem to be saying that the existing approaches have room to improve simply by training with more data. There’re also many “smart people” saying that simply having more data doesn’t turn a machine that can’t reason into one that can.

Which smart people? What is reasoning?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#469

Post by whimsicalism »

guax wrote: post #350
stephendause wrote: post #322

Earlier quoted context omitted.

I believe the relevant assertion is > Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence To me that is an open question. OpenAI has not revealed a lot of the relevant stats behind GPT-4. I also haven't seen anything about their future LLM research (though I haven't looked very hard), but I think the above assertion remains to be seen.

It seems like people are focusing on training with more data and working the data. A human does not need to learn all the contents of the internet to learn skepticism and logic. A smart Ai should probably not either, that would point me to a different approach and those are not predictable. I have no doubt we can make gpt better and more useful. I do have some doubts we can make it trustworthy enough for most of its…

We have evolutionary encoded priors - that took literally billions of years to evolve.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#470

Post by whimsicalism »

nfw2 wrote: post #222
whimsicalism wrote: post #167

Earlier quoted context omitted.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

The AI doomer speculation is of the form "if things continue improving the way they are currently improving, there is a non-zero risk that we cause human extinction." We have battle-tested rules indicating that increasing data size and compute will lower perplexity on the corpora. This is not for sure, but it is likely given inductive assumptions that things generally keep behaving the same way.

What we don't know is what qualitative capabilities are unlocked at different perplexity levels.

I think this is a lot less speculative than "things will not continue improving the way they are currently improving."

Post reply on HN