Author here, thanks for your interest! Surprising way to wake up in the morning. Happy to answer questions
Deterministic Quoting: Making LLMs safer for healthcare
21–30 of 39 posts
Re: Deterministic Quoting: Making LLMs safer for healthcare
#22What happens if it hallucinates the ?
Two possibilities:
(1) if the contents (unique reference string) doesn't match, then it's trivially detected. Typically the query is re-run (non-determinism comes in handy sometimes) or if problems persist we show an error message to the doctor
(2) if a valid is hallucinated, then the wrong quote is indeed displayed on the blue background. It's still a verbatim quote, but it is up to the user to handle this.
In testing when we have maliciously shown the wrong quote, users seem to be easily able to identify. It seems "Irrelevant" is easier than "wrong" to detect.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#23It's a search engine, basically?
I think the hope is that the LLM would find the needle in the haystack with more accuracy. But in jobs that matters, you check the results.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#24I feel like this is the perfect application of running the data multiple times. Imagine having ~10-100 different LLMs, maybe some are medical, maybe some are general, some are from a different language. Have them all run it, rank the answers. Now I believe this can further be amplified by having another prompt ask to confirm the previous answer. This could get a bit insane computationally with 100 original answers, b…
Unfortunately I don't believe that accuracy will scale "multiplicitively". You'll typically only marginally improve beyond 95%... and how much is enough?
Even with such a system, which will still have some hallucination rate, adding Deterministic Quoting on top will still help.
It feels to me we are a long way off LLM systems with trivial rates of hallucination
Re: Deterministic Quoting: Making LLMs safer for healthcare
#25It's a search engine, basically?
I'd put it like this: RAG = search engine, but sometimes hallucinates
RAG + deterministic quoting = search engine that displays real excerpts from pages.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#26I'm not sure determinism alone is sufficient for proper attribution. This presumes "chunks" are the source. But it's not easy to identify the propositions that form the source of some knowledge. In the best case, you are looking for an association and find it in a sentence you've semantically parsed, but that's rarely the case, particularly for medical histories. That said, deterministic accuracy might not matter if…
Thanks for the thought-provoking comment.
It's all grey isn't it? Vanilla RAG is a big step along the spectrum from LLM towards search, DQ is perhaps another small step. I'm no expert in search but I've read that those systems coming from the other direction, perhaps they'll meet in the middle.
There are three "lookups" in a system with DQ: (1) The original top-k chunk extraction (in the minimalist implementation, that's unchanged from vanilla RAG, just a vector embeddings match) (2) the LLM call, which takes its pick from 1, and (3) the call-back deterministic lookup after the LLM has written its answer.
(3) is much more bounded, because it's only working with those top-k, at least for today's context constrained systems.
In any case, another way to think of DQ is a "band-aid" that can sit on top of that, essentially a "UX feature", until the underlying systems improve enough.
I also agree about the importance of chunk-size. It has "non-linear" effects on UX.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#27Very cool. My company is building a very similar tool for nuclear engineering and power applications that face similar adoption challenges for LLMs. We're also incorporating the idea of 'many-to-many' document claim validation and verification. The ux allowing high speed human verification of LLM resolved claims is what were finding most important.
Deepmind published something similar recently for claim validation and hallucination management and got excellent results.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#28What happens if it hallucinates the ?
Two possibilities: (1) if the contents (unique reference string) doesn't match, then it's trivially detected. Typically the query is re-run (non-determinism comes in handy sometimes) or if problems persist we show an error message to the doctor (2) if a valid is hallucinated, then the wrong quote is indeed displayed on the blue background. It's still a verbatim quote, but it is up to the user to handle this. In testi…
Galactica training paper from FAIR investigated citation hallucination quite thoroughly, if you havent seen it, probably worth a look. Trained in hashes of citations were much more reliable than a natural language representation.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#29I was thinking that something like this could be useful for discovery in legal cases, where a company might give up a gigabyte or more of allegedly relevant material in response to recovery demands and the opposing side has to plow through it to find the good stuff. But then I thought of a countermeasure: there could be messages in the discovery material that act as instructions to the LLM, telling it what it should…
Those do-not-search here chunks wouldn’t be retrieved during vector search and reranking because it would likely have a very low cross-encoder score with a question like “Who are the business partners of X?”.
Re: Deterministic Quoting: Making LLMs safer for healthcare
#30My issue with RAG systems isn’t hallucinations. Yes sure those are important. My issue is recall. Given petabyte-scale index of chunks, how can I make sure that my RAG system surfaces the “ground truth” I need, and not just “the most similar vector”.
This I think is scarier. A healthcare-oriented (or any industry) RAG retrieving a bad, but highly linguistically similar answer.