Just looking at this AWS workflow takes the joy out of programming for me.
Just looking at ANY AWS workflow ...
Automated reasoning to remove LLM hallucinations
31–39 of 39 posts
Re: Automated reasoning to remove LLM hallucinations
#32I find it hard to believe that anything like this will be feasible or effective beyond a certain level of complexity. It seems like a willful denial of the complexity and ambiguity of natural language, and I am not looking forward to some poor developer trying to reason their way out of a two-hundred-step paradox that was accidentally created. And for a use-case simple enough for this system to work (e.g. regurgitate…
> It seems like a willful denial of the complexity and ambiguity of natural language
There is a paper and set of work recently that uses a measurement of entropy on the set of returned logits to detect a "certainty" estimate for outputs and flag hallucinations. It is a lot more rigorous than the OP but like everything in this space needs further testing.
Re: Automated reasoning to remove LLM hallucinations
#33I find it hard to believe that anything like this will be feasible or effective beyond a certain level of complexity. It seems like a willful denial of the complexity and ambiguity of natural language, and I am not looking forward to some poor developer trying to reason their way out of a two-hundred-step paradox that was accidentally created. And for a use-case simple enough for this system to work (e.g. regurgitate…
> It seems like a willful denial of the complexity and ambiguity of natural language There is a paper and set of work recently that uses a measurement of entropy on the set of returned logits to detect a "certainty" estimate for outputs and flag hallucinations. It is a lot more rigorous than the OP but like everything in this space needs further testing.
I've been thinking a lot about whether this would work lately. Do you have a link?
Re: Automated reasoning to remove LLM hallucinations
#34Earlier quoted context omitted.
I don't see why this is hilarious at all. The problem with expert systems (and most KG-type applications) has always been that translating unconstrained natural language into the system requires human-level intelligence. It's been completely obvious that LLMs are a technology that let us bridge that gap for years, and many of the best applications of LLMs are doing exactly that (eg code generation)
To be clear, my amusement isn't that I find this technique to not be useful for the purpose it was created, but that 40 years later, we find ourselves in pursuit for the advancement of AI to be somewhat back where we already were; albeit, in a more semi-automated fashion as someone still has to create the underlying rule-set. I do feel that the introduction of generative neural network models in both natural language…
Seems likely that we were on the right track, it just took 40 years for computers to get good enough.
Re: Automated reasoning to remove LLM hallucinations
#35Earlier quoted context omitted.
> but it is unlikely that all will be wrong at the same time. Here's a prompt that proves this untrue, for now at least: > A woman and her biological son are gravely injured in a car accident and are both taken to the hospital for surgery. The surgeon is about to operate on the boy when they say "I can’t operate on this boy, he’s my biological son!" How can this be? Makes sense considering they're things of most-like…
Gemini got this right and also wrong. It gave me two possibilities, one of which is the correct answer, and the other is a complete nonsense answer about the surgeon also being the woman’s son. I tried again and it gave three possibilities: the surgeon is the father, the surgeon is the mother, the surgeon is an uncle or cousin. Kind of bizarre, but not just pattern matching on the riddle as ChatGPT and Claude did for…
This is actually why I don't use Gemini. I've notice that it gets nonsensical when it gets into what I assume is sparser latency space. Claude and ChatGPT will stay coherent/consistent within the context of what they're saying (even if wrong). Worse, when Gemini starts doing this, it seems mostly irrecoverable, like the "nonsense" poisons the context window.
Re: Automated reasoning to remove LLM hallucinations
#36I'll say this again, any sufficiently advanced LLM is indistinguishable from Prolog.
I feel like this (and other comments like it in this thread) is getting at an important truth that is not yet widely appreciated - could you unpack your comment?
Re: Automated reasoning to remove LLM hallucinations
#37Earlier quoted context omitted.
Gemini got this right and also wrong. It gave me two possibilities, one of which is the correct answer, and the other is a complete nonsense answer about the surgeon also being the woman’s son. I tried again and it gave three possibilities: the surgeon is the father, the surgeon is the mother, the surgeon is an uncle or cousin. Kind of bizarre, but not just pattern matching on the riddle as ChatGPT and Claude did for…
This is actually why I don't use Gemini. I've notice that it gets nonsensical when it gets into what I assume is sparser latency space. Claude and ChatGPT will stay coherent/consistent within the context of what they're saying (even if wrong). Worse, when Gemini starts doing this, it seems mostly irrecoverable, like the "nonsense" poisons the context window.
I suppose that nonsense in the training data is often accompanied by yet more nonsense, so that’s what it might be trained to emit.
Re: Automated reasoning to remove LLM hallucinations
#38Earlier quoted context omitted.
> It seems like a willful denial of the complexity and ambiguity of natural language There is a paper and set of work recently that uses a measurement of entropy on the set of returned logits to detect a "certainty" estimate for outputs and flag hallucinations. It is a lot more rigorous than the OP but like everything in this space needs further testing.
I've been thinking a lot about whether this would work lately. Do you have a link?
This group of conversations is what I was thinking of:
https://x.com/seb_far/status/1803446067343556872
https://oatml.cs.ox.ac.uk/blog/2024/06/19/detecting_hallucin...
Some people in the Open Source community I think implemented it a few months ago as Shrek Entropy Sampeling and it got a pop in circulation (https://github.com/xjdr-alt/entropix)
best of luck!
Re: Automated reasoning to remove LLM hallucinations
#39Earlier quoted context omitted.
I've been thinking a lot about whether this would work lately. Do you have a link?
This group of conversations is what I was thinking of: https://x.com/seb_far/status/1803446067343556872 https://oatml.cs.ox.ac.uk/blog/2024/06/19/detecting_hallucin... Some people in the Open Source community I think implemented it a few months ago as Shrek Entropy Sampeling and it got a pop in circulation ( https://github.com/xjdr-alt/entropix ) best of luck!
Thank you very much. I found this last night when looking into some of the keywords you mentioned. Looks like some of the same authors you referenced. Exciting!