Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teach your LLM to answer with facts, not fiction

blog.myscale.com

Post reply on HN
Print view View original

131–140 of 149 posts

Re: Teach your LLM to answer with facts, not fiction

#131

Post by SkyPuncher »

NoZebra120vClip wrote: post #39

You know, aside from this being a blatant feature-length advertisement for what they're selling, I almost thought this was a clever idea. I thought it involved prompting the LLM to write SQL code to query a knowledge base of documents, and index into them, so that you'd know where to look in the original documents for your authoritative answer. So it would be a meta-search agent. But apparently, they intend the queri…

I cannot figure out why LLMs are relevant to their solution. This whole thing comes down to a similarity search via vectors.

The LLM layer seems completely unnecessary. Why do you have a schema that requires an LLM to decide which column to query (which is the LLM's only unique value in this proposal)? Why are you not normalizing into a single column?

Re: Teach your LLM to answer with facts, not fiction

#132

Post by SkyPuncher »

senectus1 wrote: post #10

would be nice if they could show a gradient score on results that show how certain it is of its answers... It should be fairly trivial for it to tell you how often its straight up lied about something.

It's essentially impossible. Confidence is often a situational measure.

"Is the sky blue?"

* Generally, yes. If you ask a 5 year old, the answer is yes.

* Is the sky blue right now? Maybe, maybe not. You need to look outside. Even then, you might have wild fire haze. Is it still blue? Is it orange? When does blue become orange?

* Is the sky blue in Blade Runner? Doesn't really seem like it.

-----

Further, who are you talking to? Is this trivia night where your best guess is better than no guess? Is this a scientific panel? Do you have alternative options? Do those alternative options align with your opinion? If you're wrong, how wrong are you?

Re: Teach your LLM to answer with facts, not fiction

#133

Post by astrange »

FishInTheWater wrote: post #108
astrange wrote: post #92

Earlier quoted context omitted.

What they are doing depends entirely on what decoding algorithm you use. An LLM is mostly a token probability function, but it's not just that - a transformer model is capable of learning anything. Tokens are the interface, not necessarily the implementation.

A transformer can only memorize , it doesn't learn to do . For what that concerns us here: LLMs will never learn to fact-check anything. They'll blindly regurgitate the facts they have been "taught", but never consider or evaluate "the paper cited for this fact on wikipedia is a bunch of bullshit". Any attempt to use them to produce "facts" is ultimately just folly, in the same way Google's attempt to do so with it's…

No, a transformer is a universal function approximator and is capable of learning to do anything to some degree of accuracy.

GPT doesn't do math correctly but it also doesn't just memorize it.

Re: Teach your LLM to answer with facts, not fiction

#134

Post by ajross »

FishInTheWater wrote: post #126
ajross wrote: post #123

Earlier quoted context omitted.

> [LLMs] never consider or evaluate "the paper cited for this fact on wikipedia is a bunch of bullshit". Nor do people, though! This is setting the bar way too high. The whole point to having edited reference sources like "encyclopedias" is that so that we can rely on the expertise of the editors in lieu of having to develop the expertise ourselves[1]. No, an LLM that simply knows a priori (via prompt hacking) which…

This is setting the bar way too high. No. If these things are claimed to be sources of truth, then the bar needs to be that high. It is precisely because people don't fact-check that the bar has to be so high.

> If these things are claimed to be sources of truth

That's a strawman, though. No service, nor human, "claims to be a source of truth" in the kind of profound sense you seem to be using. It stops, everywhere, at "Wikipedia (or whatever) said it and I trust it".

The only way to get access to deeper expertise is to (1) BE an expert and (2) engage in an discussion with another.

Re: Teach your LLM to answer with facts, not fiction

#135

Post by mjburgess »

Ajedi32 wrote: post #129
mjburgess wrote: post #119

Earlier quoted context omitted.

Here, "supposed to guess" means "having the goal of..." So no LLM knows what it's supposed to do. If you prefer, you could say it only ever has one goal: to generate a sequence of tokens which are jointly the most probable to occur along with the prompt tokens, given such probabilities in a historical corpus. This imitates knowledge, goal-directness, "inferring context" etc. without doing any of those things. Conside…

What does RLHF do then? I feel like you completely ignored the central point of GP's comment. RLHF is the difference between GPT-3.5 and ChatGPT, and it's the whole reason why LLMs are suddenly such a big deal. ChatGPT demonstrated that it's possible to give language models a goal beyond just "complete most likely next word" and that they can actually be somewhat competent at achieving those goals despite not being e…

> competent at achieving those goals despite not being explicitly trained for them.

Well (1) it doesn't achieve goals, since a "goal" is observer-relative. We have goals, the LLM has a formal optimisation objective which gives it the appearence of goal-directed behaviour (in a similar way, eg., that it appears pens want to fall when dropped).

And (2), reading your "goal" here even in observer-relative ways, I don't think there's much evidence of this. These models are "trained" on everything ever written, include all of the internet and basically all digitised book. I don't see any evidence of much generalisation -- if you can find it by google, then the LLM has it stored compressed (ie., the "weights").

The innovation in LLMs is being able to compute `max P(answer|prompt, historical_corpus)` for increasingly longer prompts --- there's no innovation in goal-directed behaviour.

That's VC propangada to disguise the fact that LLMs are mostly an innovation in copyright laundering.

Re: Teach your LLM to answer with facts, not fiction

#136

Post by yifanl »

tyre wrote: post #125
jameshart wrote: post #22

'Facts' aren't as black and white as people think. "What does Charmander evolve into?" "What does the spell 'avada kedavra' do?" "What is the Sindarin word for 'friend'?" "What are the names of Santa's reindeer?" "Where did Robin Hood live?" "Where did Achilles die?" These are all 'factual questions' you can find answers to from reputable sources like Wikipedia. Google displays 'fact boxes' for several of them. Wolfr…

> What does Charmander evolve into? wait why is this implied to not be black and white? Charmeleon is the only correct answer.

Charmanders don't evolve into anything, it doesn't exist in the natural world.

Re: Teach your LLM to answer with facts, not fiction

#137

Post by yifanl »

RyEgswuCsn wrote: post #68

I think LLMs need to be taught to say "I don't know"/"I am not sure" or something to that effect. Another approach might be to introduce an adversarial "censor" model to guard against hallucination (or inappropriate answers).

That sounds like admitting there is no secret sauce?

The superpower has been the ability to synthesize output from very disparate training sources, and any answer for "I don't know" would come from needing to synthesize disparate training sources.

Re: Teach your LLM to answer with facts, not fiction

#139

Post by jameshart »

sweezyjeezy wrote: post #105
jameshart wrote: post #22

'Facts' aren't as black and white as people think. "What does Charmander evolve into?" "What does the spell 'avada kedavra' do?" "What is the Sindarin word for 'friend'?" "What are the names of Santa's reindeer?" "Where did Robin Hood live?" "Where did Achilles die?" These are all 'factual questions' you can find answers to from reputable sources like Wikipedia. Google displays 'fact boxes' for several of them. Wolfr…

> it doesn't know if it's supposed to guess a probable word from a Wikipedia article, an Onion article, a Project Gutenberg manuscript, or an Archive Of Our Own fanfic. So you get a bit of all that. This is true of base LLM models that are just trained on missing-word prediction on the training corpus, but one of the main points of RLHF[1] is to tune this model to make these kind of inferences the way a human would e…

Right.

Reinforcement learning trains them that question and answer sessions contain answers which statistically correlate with factual statements in their broader learning corpus.

When formulating answers, this leads them to formulate answers that reflect the factual information on which they were trained.

My point is that the source data contains a far muddier range of information than just unarguable facts.

We largely want LLM based Q&A bots to answer questions about fictional or mythical characters in their own terms. As I said, those questions above all have reasonably ‘correct’ answers.

The fact that from all that LLMs do as well as they do is remarkable. But it also seems like it requires us to assume that LLMs are capable of a remarkable degree of cultural nuance, media literacy and contextual awareness for them to figure out the different authorship, salience, trustworthiness, agenda, biases, and assumptions of all the gigareams of text they’ve ingested.

Re: Teach your LLM to answer with facts, not fiction

#140

Post by jameshart »

immibis wrote: post #55
xyzzy123 wrote: post #45

Earlier quoted context omitted.

I kinda like this but e.g are research papers fact or fiction? How about an economics textbook, or an article in the economist? "A history of the english speaking peoples" by Winston Churchill? If we restrict to "ground truth we feel very sure about" it feels like available training data might be quite small.

and what if the economics textbook contains "much like Charmander evolves into Charizard, free markets evolve into monopolies"?

Right. A lot of the magic of LLMs probably comes from the broader appreciation of language and cultural reference that they get from being trained on a diverse corpus, rather than just a bunch of dictionaries and reference books.

And anyway - answers to all my ‘fictional facts’ questions above can be sourced from Wikipedia - there’s tons of made up stuff on there.

Post reply on HN