Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

giskard.ai

Post reply on HN
Print view View original

71–80 of 85 posts

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#71

Post by kolinko »

MuffinFlavored wrote: post #55
fatso784 wrote: post #16

Looks a bit like snakeoil to me. A lot of companies now spinning up simple demos with opaque backends, making huge claims they’ve solved X hard problem for/with AI, then saying “trust us” and “join our waitlist” without hard details or facts to show for it. If you could detect hallucinations/biases etc that easily, don’t you think OpenAI would’ve worked on something like this?

> don’t you think OpenAI would’ve worked on something like this? Along this line of thought: was it a massive oversight for them to not train the model to say "math detected, let me pass that to a solver" instead of trying to guess what token should come next in a math problem?

They are solving that with plugins now.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#72

Post by mattbit »

krona wrote: post #50
henri18 wrote: post #44

Earlier quoted context omitted.

There are many ways to detect hallucinations. Basically, either you have the ground truth answers in external database, in that case you compare to ground truths. Or you don’t have the ground truth. In that case, you need to do metamorphic testing. See this article on it: https://www.giskard.ai/knowledge/how-to-test-ml-models-4-met...

Metamorphic testing seems to try to map an output of a model to a ground truth, which I guess is great if you have a database of all the known truths in the universe.

Not exactly, metamorphic testing does not need an oracle. That’s actually the reason of its popularity in ML testing. It works by perturbing the input in a way that will produce a predictable variation of the output (or possibly no variation).

Take for example a credit scoring model: you can reasonably expect that if you increase the liquidity, the credit score should not decrease. In general it is relatively easy to come up with a set of assumptions on the effect of perturbation, which allows evaluating the robustness of a model without knowing the exact ground truth.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#74

Post by danShumway »

fatso784 wrote: post #16

Looks a bit like snakeoil to me. A lot of companies now spinning up simple demos with opaque backends, making huge claims they’ve solved X hard problem for/with AI, then saying “trust us” and “join our waitlist” without hard details or facts to show for it. If you could detect hallucinations/biases etc that easily, don’t you think OpenAI would’ve worked on something like this?

In this area, if there's not a public demo and the results aren't verifiable, then it's not worth paying attention to.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#75

Post by danShumway »

Googleton wrote: post #27
anonzzzies wrote: post #3

How does this work? Does anyone know? And for a large swats of things, how can it possibly work? It’s not possible to say if or if not it is hallucinating code for almost all code and apis, for instance. And I see similar issues with many fields outside pure facts. With privacy issues as well.

We are a team of engineers & researchers on AI Alignment & Safety, we're investigating multiple methods, including metamorphic testing, human feedback, benchmarks with external data sources, and LLM explainability methods. Currently, fact checking works on straight facts. It does a Google Search and uses LLMs to shorten it. Once it has the short version, it will compare the short results with the answer provided by C…

> Currently, fact checking works on straight facts. It does a Google Search and uses LLMs to shorten it.

So your fact-checking LLM is also vulnerable to injection and unethical prompting then when it ingests website text. And a Google search is far, far away from fact checking, particularly for the subtle errors that GPT-4 is prone to making.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#76

Post by some_random »

crop_rotation wrote: post #49
henri18 wrote: post #44

Earlier quoted context omitted.

There are many ways to detect hallucinations. Basically, either you have the ground truth answers in external database, in that case you compare to ground truths. Or you don’t have the ground truth. In that case, you need to do metamorphic testing. See this article on it: https://www.giskard.ai/knowledge/how-to-test-ml-models-4-met...

But GPT4 doesn't hallucinate on things which are popular enough to be replicated enough times on the web as knowledge. It hallucinates on things which are very less likely to be repeated many times. That rules out an external database with true answers. Unless the external database is supposed to contain all info queryable in all ways, in which case the database is just a better version of GPT-X. The metamorphic test…

I've been playing with GPT4 summarization of hard knowledge that has an external database with true answers that GPT knows about, and it's still hallucinating regularly.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#77

Post by bbwbsb »

aftbit wrote: post #57
plagiarist wrote: post #7

Earlier quoted context omitted.

Such as?

There were lots of good examples during COVID. Remember that you don't need to wear masks, because washing your hands is enough (and we need to save the masks for doctors, but we're afraid to say that, because it will cause a run on masks). Remember that staying 6 feet apart is a magic distance over which COVID cannot cross (or maybe 1 meter, truth depends on the country you live in). Remember that you can't eat insi…

A big difficulty is the conflation of fact with judgement. 'Vaccines work', 'masks don't work', 'a lab leak is impossible', etc. are judgements, not facts. They are not even hypotheses, in that there is no clear criteria by which they can be falsified. Hence fact-checking presents obvious problems, as in practice it will be judgement-checking.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#78

Post by kmod »

fatso784 wrote: post #16

Looks a bit like snakeoil to me. A lot of companies now spinning up simple demos with opaque backends, making huge claims they’ve solved X hard problem for/with AI, then saying “trust us” and “join our waitlist” without hard details or facts to show for it. If you could detect hallucinations/biases etc that easily, don’t you think OpenAI would’ve worked on something like this?

I think part of the problem is that it's technically correct to say "my product does X" even if it does X extremely poorly. I'm not sure if this can be changed because any line for "does-X vs does-not-adequately-do-X" is going to necessarily be subjective.

So personally I think the problem is that people see "this product does X" and interpret that to mean that it does X well. I don't think it's necessarily bad that we're seeing an explosion of AI tools that are a bit underwhelming if people understood it as such -- we're on, after all, a site with a heavy startup focus and saying "your product doesn't do everything that I want" is a bit antithetical to that.

But yeah specifically for this one there are arguments that "X is not even possible, especially not with this approach" so it's a bit more egregious.

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#79

Post by BoorishBears »

samstave wrote: post #59
flangola7 wrote: post #58

Earlier quoted context omitted.

OpenAI plugins can connect with a growing number of things. Zapier is one, which is already several thousand functions.

Can you give some example recipes?

Most obvious one is Wolfram, it passes most math to Wolfram

Re: SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

#80

Post by sebzim4500 »

thfuran wrote: post #42
sebzim4500 wrote: post #6

Seems too good to be true, and I don't understand what it even means for an LLM to be unbiased.

One with only weights and no biases in the ANN is unbiased.

Someone I know thought that LLaMA was unbiased because they 'read' the paper and clearly didn't know what anything meant. A great example of "a little knowledge is a dangerous thing".
Post reply on HN