Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Truth is not a direction: a Tarski attack on LLM probes

abeljansma.nl

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Truth is not a direction: a Tarski attack on LLM probes

#41

Post by maxbond »

elendilm wrote: post #25
voxl wrote: post #16

Earlier quoted context omitted.

A basic course in statistics will inform you of why a 99.99% accurate test should be looked at with skepticism when diagnosing a rare disease. Yet we see the fancy 9s and think somehow this many 9s is enough.

Really? Laws of probability works against your argument. Such a basic course in statistics needs to be scrutinized.

If 1 in 10,000 people have a disease, then a "test" which always reports the patient doesn't have the disease will be correct 99.99% of the time. "99.99% accuracy" should be "looked at with skepticism" in that it doesn't tell you what you need to know to understand the quality of a a test for a rare disease (a classifier under conditions of severe class imbalance); at a minimum, you would want to understand it's false positive and false negative rate, not (just) it's overall error rate.

See example "A": https://en.wikipedia.org/wiki/Base_rate_fallacy

Re: Truth is not a direction: a Tarski attack on LLM probes

#42

Post by FeepingCreature »

Terr_ wrote: post #18
Legend2440 wrote: post #4

I think this article pushes the premise farther than is reasonable. The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.

> The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. I think even that's too-optimistic: The LLM is a document-extender, so its "belief" is whether a token seems like it would statistically fit-next in a partial document, based on prior documents. This is usually not the kind of analytic truth we're interested in, and we've already figure…

I half-disagree: this is exactly the kind of analytic truth humans are usually interested in. "Truth" as perceived by humans is based on a massive system of prefiltering, narrowing, preprocessing and situational awareness.

A drop of water falls on your hand. Is it raining? Depends. Are you painting a watercolor picture outside? Then probably yes. Or are you going to the store? Then probably no. So truth is inseparable in practice from usecase. Is a whale a fish? I don't know, are you a geneticist or a poet? It's all mood and usecase. I'm not convinced there's any difference in kind between the LLM's speaker-selection and a human's choice of research field.

That is to say, it's not that I think you're wrong, it's that there is no other pursuit of truth than what you describe. If anything, the LLM's pursuit of next-token prediction is unusually honest for a truth-seeker.

Re: Truth is not a direction: a Tarski attack on LLM probes

#43

Post by FeepingCreature »

Vecr wrote: post #3

Someone at MIRI must know how to solve this. Good luck getting them to tell you how!

For context: Logical Induction https://arxiv.org/abs/1609.03543

Sadly the line of research seems to have been abandoned. Kind of understandable with the ascent of LLMs: there is no time left for a multi-decade research program.

Re: Truth is not a direction: a Tarski attack on LLM probes

#44

Post by baq »

lstodd wrote: post #40
baq wrote: post #28

Title is a bit clickbaitish, but the content is well worth reading - came in with my pitchfork ready and left agreeing with basically all of it, with questions like ‘what if the probe could return 3 dimensions: truthfulness, knowledge confidence and decidability?’ Also the observation that people treat LLMs like oracles when they’re everything but is spot on, something I’ve also been thinking about and it’s quite a b…

> what if the probe could return 3 dimensions: truthfulness, knowledge confidence and decidability? can't be done because the other two dimensions depend on the first. or in other words it's all the same dimension only with a different name.

'I know this is true' is different than 'This is true' is different than 'It is impossible to say if it's true or false' is different than 'I've no idea', right?

Re: Truth is not a direction: a Tarski attack on LLM probes

#45

Post by TeMPOraL »

smallnix wrote: post #38
kstenerud wrote: post #21

> It might seem absurd to you to even suggest superhuman AIs could function as a truth-oracle (it certainly does to me), but there are two reasons to take it seriously. First, it is how these things will be used practically by the vast majority of people. They are already replacing standard Google search results, and I’ve had many discussions end with people delegating final authority on the truth to an AI. There are…

> The set-up assumes that the game and life are the same thing, and such is the pervasive nature of the idea of the game within the society that just by believing that, they make it so. (The player of Games, banks)

Not sure what you're quoting, but what the quote seems to describe is known by other names, including "intersubjectivity", and is the very thing on which all our society and civilization stands. "Family", "loyalty", "society", "money", "market", "government", "law", "limited liability corporations", etc. are all abstract ideas made real through power of shared belief.

More specifically (and this is IMO critical), what pins such concepts down and makes them real, is not belief in the concept - but everyone's belief that everyone else believes in those concepts. E.g. it's not my belief in value of money that makes money valuable for me - it's my belief that the bank and the shopkeeper and the taxman all believe the money is worth something, which they do because they believe everyone else agrees too. This recursion allows us to expect certain things, such as me expecting to get food from store in exchange for money, and as long as enough people believe everyone else believes, the concept is as good as real.

Point being: this is not some gotcha or trickery, but the very stuff our lived reality is made off.

Re: Truth is not a direction: a Tarski attack on LLM probes

#46

Post by Xmd5a »

H: All right, all right... and the limitation of human intelligence?

M: Mr Gödel is telling you that the theorem limits formal systems. So you see your intelligence as a formal system, as a machine?

H (pompously): Indeed. I have the impression that everything I do ought to be done by a machine, which could moreover speak just as well in my place.

G: From where I am, it is difficult to tell whether you exist or whether you are the virtual creation of a GAT - a Generator of Automatic Truisms. Intelligence does not exist without error, perhaps even without obstinacy in error; but who would take the risk of giving a computer that kind of psychology? As for the incompleteness theorem, it certainly did not foresee bad-tempered theories...

---

Gödel's Theorem, or an Evening with Mr Homais Jean-Yves Girard

Translation: https://files.catbox.moe/kac0wu.pdf

Original: https://perso.ens-lyon.fr/pierre.lescanne/ENSEIGNEMENT/LOGIQ...

Re: Truth is not a direction: a Tarski attack on LLM probes

#47

Post by TeMPOraL »

mike_hearn wrote: post #39
Legend2440 wrote: post #4

I think this article pushes the premise farther than is reasonable. The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.

Not only that, but nobody really cares about whether truth vectors can work correctly under carefully constructed paradox edge cases. Well, maybe mathematicians do, but nobody else.

Mathematicians, philosophers, cognitive scientists, and others.

The latent space encoded in the weights of a model is itself an object of study, just as interesting as the model itself.

Re: Truth is not a direction: a Tarski attack on LLM probes

#48

Post by TeMPOraL »

Terr_ wrote: post #18
Legend2440 wrote: post #4

I think this article pushes the premise farther than is reasonable. The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.

> The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. I think even that's too-optimistic: The LLM is a document-extender, so its "belief" is whether a token seems like it would statistically fit-next in a partial document, based on prior documents. This is usually not the kind of analytic truth we're interested in, and we've already figure…

Worth remembering what the goal function behind the next token prediction is. It's what makes it go beyond moods and styles, and work with concepts of fact the same way we do.

Re: Truth is not a direction: a Tarski attack on LLM probes

#49

Post by elendilm »

maxbond wrote: post #41
elendilm wrote: post #25

Earlier quoted context omitted.

Really? Laws of probability works against your argument. Such a basic course in statistics needs to be scrutinized.

If 1 in 10,000 people have a disease, then a "test" which always reports the patient doesn't have the disease will be correct 99.99% of the time. "99.99% accuracy" should be "looked at with skepticism" in that it doesn't tell you what you need to know to understand the quality of a a test for a rare disease (a classifier under conditions of severe class imbalance); at a minimum, you would want to understand it's fals…

If only stating the obvious was a fallacy :)

You appear not to have understood probability theory my friend. You will never get 100% in this universe for anything. What if "its a simulation" or "a dream" arguments ensures you never acheive 100%.

Bayes probability theory will be a good start for you.

https://en.wikipedia.org/wiki/Bayes%27_theorem

Re: Truth is not a direction: a Tarski attack on LLM probes

#50

Post by TeMPOraL »

zarzavat wrote: post #10
TZubiri wrote: post #7

Earlier quoted context omitted.

If anything, the whole vector space is the LLM's truth.

LLMs are not optimized only for truth they are optimized for a more complicated objective that includes e.g. humans liking their output. It is a universal truth that to get humans to like you, you have to lie to them.

Yes. Much like our own communication. Thus is the difference between a research paper and the poem. People like the former for objectivity, the latter for beauty.
Post reply on HN