Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTZero Case Study – Exploring False Positives

gonzoknows.com

Post reply on HN
Print view View original

71–80 of 120 posts

Re: GPTZero Case Study – Exploring False Positives

#71

Post by elefanten »

dr_dshiv wrote: post #37
brookst wrote: post #10

Earlier quoted context omitted.

The best was the Ted Chiang article making numerous category errors and forest/trees mistakes in arguing that LLMs just store lossy copies of their training data. It was well-written, plausible, and so very incorrect.

I felt the same way. But I’d love to read a specific critique. Have you seen one?

Here’s one from a researcher (which also links to another), though I’m not qualified to assess it’s content in depth.

https://twitter.com/raphaelmilliere/status/16240731504754319...

Re: GPTZero Case Study – Exploring False Positives

#72

Post by dsign »

FPGAhacker wrote: post #62
dsign wrote: post #34

I had some fun yesterday when ChatGPT hallucinated a bibliographic reference to an article that didn't exist. But the journal existed, and it had plenty of articles that made ChatGPT's hallucination plausible. I think that at least this use case can be fixed with some pragmatic engineering[^1]. [^1]: Which may take a bit to happen, because our current crop of AI researchers have all taken "The bitter lesson"[^2] to h…

Did chatgpt post the reference as a footnote (or parenthetical)? At least for now,I was thinking it didn’t do that, and maybe the lack of references would be an indicator of unedited gpt output.

It did:

...

> However, according to a study published in the Journal of Dairy Science, the diacetyl content of butter can range from approximately 0.5 to 14 parts per million (ppm) (https://doi.org/10.3168/jds.S0022-0302(03)73775-3).

The doi I could not find, so I'm pretty sure is bogus.

I asked it to produce a full reference:

> Sure, the full reference for the study I mentioned is:

> Yvon, M., Chambellon, E., & Bolotin, A. (2003). Effect of pH on diacetyl and acetoin production by Lactococcus lactis subsp. lactis biovar diacetylactis. Journal of dairy science, 86(12), 4068-4076.

I went to the index of the journal 86(12), and that article is not there.

Re: GPTZero Case Study – Exploring False Positives

#73

Post by jldugger »

sigmoid10 wrote: post #6
Maken wrote: post #2

Alternative title: Most academic papers are indistinguishable from AI generated babble.

...to AI. It's kinda funny how this is yet another area where these models suck very much in the same way that most humans do. LLMs are bad at arithmetic? So are most people. Can't tell science from babble? I already wouldn't ask a non-expert to rate any aspect of an academic paper. Trusting the average Joe who has only completed some basic form of education would be tremendously stupid. Same with these models. Maybe…

>...to AI.

Perhaps we can call it the "Synthromorphic principle," the bias of AI agents to project AI traits onto conversants that are not in fact AI.

Re: GPTZero Case Study – Exploring False Positives

#74

Post by sdflhasjd »

gtsnexp wrote: post #61

Detecting text generated by large language models like ChatGPT is a challenging task. One of the main difficulties is that the generated text can be highly variable and can cover a wide range of topics and styles. These models have learned to mimic human writing patterns and can produce text that is grammatically correct, semantically coherent, and even persuasive, making it difficult for humans to distinguish betwee…

Ironic, because to me this was so obviously ChatGPT to me from literally the first sentence.

I think this is probably because it doesn't match the conversational style of a forum discussion.

Re: GPTZero Case Study – Exploring False Positives

#75

Post by theptip »

Closi wrote: post #52
sebzim4500 wrote: post #44

Earlier quoted context omitted.

>detection isnt that difficult now I would have thought this, but every attempt I've seen at detecting chatGPT generated text has failed miserably.

It fails on false positives, but you don't often get false negatives, which might be enough at the moment for quite a few use-cases. Also false positives are typically "this text is likely to contain parts that were AI generated" rather than "This text is higly likely to be AI generated" (which is what GPT-generated content generally produces). When I've tried to prompt-engineer GPT to produce text that GPTZero will…

Of course false positives matter; the naive heuristic “everything is AI generated” has zero false negatives, and mostly false positives. In the OP half of the positives are false. That’s not a useful signal IMO. You couldn’t use that to police homework for example.

Re: GPTZero Case Study – Exploring False Positives

#76

Post by flir »

valine wrote: post #60
RC_ITR wrote: post #45

Earlier quoted context omitted.

> Accuracy is obviously going to improve in models Well, to be clear, they can put rules based filters and other things on top of the neural net, but the core GPT will never get more accurate since it has no mechanism to understand what words mean.

GPT3 is far more accurate than GPT2. Seems reasonable that larger models trained on more data will continue to improve accuracy. I'd also expect larger models to be better at summarizing text, ie potentially fixing the Bing issues where it hallucinates numbers. Our models sizes are a product of our scaling and hardware limitations. There's no reason to believe we are anywhere near optimal.

> Seems reasonable that larger models trained on more data will continue to improve accuracy.

It also seems reasonable to assume that they will eventually encounter diminishing returns, and that the current issues, such as hallucinations, are inherent to the approach and may never be resolved.

To be clear I don't have a clue which statement is true (though I don't see why scaling would solve the hallucination problem).

Re: GPTZero Case Study – Exploring False Positives

#77

Post by moffkalast »

howlin wrote: post #30
brookst wrote: post #16

Earlier quoted context omitted.

One of the big complaints with LLMs is the confident hallucination of incorrect facts, like software APIs that don’t exist. But the way I see it, if ChatGPT thinks the Python list object should have a .is_sorted() property, that’s a pretty good indication that maybe it should. I work in PM (giant company, not Python), and one of these days my self-control will fail me and I will open a bug for “product does not suppo…

> LLMs is the confident hallucination of incorrect facts This is a very common feature of delirium in people. Chatting with an LLM seems a lot like what it would be to talk to a clever person with encyclopedic knowledge, who is just waking up from anaesthesia or is sleep talking.

Or just the average person on reddit's r/confidentlyincorrect.

Re: GPTZero Case Study – Exploring False Positives

#79

Post by robocat »

mtlmtlmtlmtl wrote: post #68
martinflack wrote: post #50

Earlier quoted context omitted.

It's probably a short-term social phenomenon. We don't bother detecting mathematical output from calculators or spreadsheets; we just like that folks give us the right answer, even if they had easy tooling to produce it. However, watching someone do things the old way would seem bemusing. If you watched a manager notating all over a physical spreadsheet with a pencil (as was commonly done at one time) it would seem q…

"Write an email that says I did X and they should do Y, but if Z then W, and we should schedule a meeting with P and Q." I feel like for most emails I write, information density is close to a maximum. This means there's no actual gain to be had from a language model. The email I would write myself is going to be about the same length as the prompt I'd have to write anyway.

There is a huge gain if English is your second language, and you use ChatGPT to rewrite, or translate.

Even if English is your mother tongue, if your written English is crappy or you need to write in a style you are unfamiliar with (e.g. formal), then ChatGPT can help.

Re: GPTZero Case Study – Exploring False Positives

#80

Post by scotty79 »

wand3r wrote: post #12

I saw this[1] interview with Sam Altman touching on interim AI impact. I really agree with his point that basically detecting output from LLMs is basically going to be futile and only really relevant in the near term. Accuracy is obviously going to improve in models and detection isnt that difficult now but will be in the future, especially if output is modified or an attempt to obfuscate origin is made. [1] https://…

I hope gpt detectors will evolve into general bs detectors.
Post reply on HN