Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTZero Case Study – Exploring False Positives

gonzoknows.com

Post reply on HN
Print view View original

61–70 of 120 posts

Re: GPTZero Case Study – Exploring False Positives

#61

Post by gtsnexp »

Detecting text generated by large language models like ChatGPT is a challenging task. One of the main difficulties is that the generated text can be highly variable and can cover a wide range of topics and styles. These models have learned to mimic human writing patterns and can produce text that is grammatically correct, semantically coherent, and even persuasive, making it difficult for humans to distinguish between the text generated by machines and the ones written by humans.

Another challenge is that large language models are highly complex and constantly evolving. GPT-3, for example, was trained on a massive dataset of text and can generate text in over 40 languages. With this level of complexity, it can be challenging to develop detection systems that can keep up with the ever-changing text generated by these models.

To implement a reliable detection system like GPTZero, which is designed to detect text generated by GPT-3, several challenges need to be addressed. First, the system needs to be highly accurate and efficient in identifying text generated by GPT-3. This requires a deep understanding of the underlying language model and the ability to analyze the text at a granular level.

Second, the system needs to be scalable to handle the vast amounts of data generated by GPT-3. The detection system should be able to analyze a large volume of text in real-time to identify any instances of generated text.

Finally, the system needs to be adaptable to the evolving nature of large language models. As these models continue to improve and evolve, the detection system needs to keep up and adapt to the changing landscape.

Re: GPTZero Case Study – Exploring False Positives

#62

Post by FPGAhacker »

dsign wrote: post #34

I had some fun yesterday when ChatGPT hallucinated a bibliographic reference to an article that didn't exist. But the journal existed, and it had plenty of articles that made ChatGPT's hallucination plausible. I think that at least this use case can be fixed with some pragmatic engineering[^1]. [^1]: Which may take a bit to happen, because our current crop of AI researchers have all taken "The bitter lesson"[^2] to h…

Did chatgpt post the reference as a footnote (or parenthetical)?

At least for now,I was thinking it didn’t do that, and maybe the lack of references would be an indicator of unedited gpt output.

Re: GPTZero Case Study – Exploring False Positives

#63

Post by ericmcer »

gtsnexp wrote: post #61

Detecting text generated by large language models like ChatGPT is a challenging task. One of the main difficulties is that the generated text can be highly variable and can cover a wide range of topics and styles. These models have learned to mimic human writing patterns and can produce text that is grammatically correct, semantically coherent, and even persuasive, making it difficult for humans to distinguish betwee…

It was weird how easy this was to identify if you have read any amount of ChatGPT content. It has a particular writing style that is pretty obvious.

I am not sure how you would code something to detect an author based on writing style. It feels like something people would have tried to do before. Probably using a similar approach that LLMs use but with a separate predictor for specific authors.

Re: GPTZero Case Study – Exploring False Positives

#64

Post by [deleted] »

ericmcer wrote: post #63
gtsnexp wrote: post #61

Detecting text generated by large language models like ChatGPT is a challenging task. One of the main difficulties is that the generated text can be highly variable and can cover a wide range of topics and styles. These models have learned to mimic human writing patterns and can produce text that is grammatically correct, semantically coherent, and even persuasive, making it difficult for humans to distinguish betwee…

It was weird how easy this was to identify if you have read any amount of ChatGPT content. It has a particular writing style that is pretty obvious. I am not sure how you would code something to detect an author based on writing style. It feels like something people would have tried to do before. Probably using a similar approach that LLMs use but with a separate predictor for specific authors.

[deleted]

Re: GPTZero Case Study – Exploring False Positives

#65

Post by mds »

ericmcer wrote: post #63
gtsnexp wrote: post #61

Detecting text generated by large language models like ChatGPT is a challenging task. One of the main difficulties is that the generated text can be highly variable and can cover a wide range of topics and styles. These models have learned to mimic human writing patterns and can produce text that is grammatically correct, semantically coherent, and even persuasive, making it difficult for humans to distinguish betwee…

It was weird how easy this was to identify if you have read any amount of ChatGPT content. It has a particular writing style that is pretty obvious. I am not sure how you would code something to detect an author based on writing style. It feels like something people would have tried to do before. Probably using a similar approach that LLMs use but with a separate predictor for specific authors.

ChatGPT in particular writes in middle school essay format: introduction, point 1, point 2, point n, conclusion.

Re: GPTZero Case Study – Exploring False Positives

#67

Post by Maken »

sigmoid10 wrote: post #6
Maken wrote: post #2

Alternative title: Most academic papers are indistinguishable from AI generated babble.

...to AI. It's kinda funny how this is yet another area where these models suck very much in the same way that most humans do. LLMs are bad at arithmetic? So are most people. Can't tell science from babble? I already wouldn't ask a non-expert to rate any aspect of an academic paper. Trusting the average Joe who has only completed some basic form of education would be tremendously stupid. Same with these models. Maybe…

I do agree that the most likely reason is that scientific papers tend to be highly formulaic and follow strict structures, so a LLM is be able to generate something much more alike to human writing than if it tries to generate narrative.

But it's still fun to deduce that the reason is that the quality of technical writing has sunk so low, that is even below the standards for AI generated text.

Re: GPTZero Case Study – Exploring False Positives

#68

Post by mtlmtlmtlmtl »

martinflack wrote: post #50
wand3r wrote: post #12

I saw this[1] interview with Sam Altman touching on interim AI impact. I really agree with his point that basically detecting output from LLMs is basically going to be futile and only really relevant in the near term. Accuracy is obviously going to improve in models and detection isnt that difficult now but will be in the future, especially if output is modified or an attempt to obfuscate origin is made. [1] https://…

It's probably a short-term social phenomenon. We don't bother detecting mathematical output from calculators or spreadsheets; we just like that folks give us the right answer, even if they had easy tooling to produce it. However, watching someone do things the old way would seem bemusing. If you watched a manager notating all over a physical spreadsheet with a pencil (as was commonly done at one time) it would seem q…

"Write an email that says I did X and they should do Y, but if Z then W, and we should schedule a meeting with P and Q."

I feel like for most emails I write, information density is close to a maximum. This means there's no actual gain to be had from a language model. The email I would write myself is going to be about the same length as the prompt I'd have to write anyway.

Re: GPTZero Case Study – Exploring False Positives

#69

Post by supriyo-biswas »

brookst wrote: post #10
sigmoid10 wrote: post #6

Earlier quoted context omitted.

...to AI. It's kinda funny how this is yet another area where these models suck very much in the same way that most humans do. LLMs are bad at arithmetic? So are most people. Can't tell science from babble? I already wouldn't ask a non-expert to rate any aspect of an academic paper. Trusting the average Joe who has only completed some basic form of education would be tremendously stupid. Same with these models. Maybe…

The best was the Ted Chiang article making numerous category errors and forest/trees mistakes in arguing that LLMs just store lossy copies of their training data. It was well-written, plausible, and so very incorrect.

Neural network based compression algorithms[1] are a thing, so I believe Ted Chiang's assessment is right. Memorization (albeit lossy) is also how the human brain works and develops reasoning[2].

[1] https://bellard.org/nncp/

[2] https://www.pearlleff.com/in-praise-of-memorization

Re: GPTZero Case Study – Exploring False Positives

#70

Post by [deleted] »

brookst wrote: post #16
panarky wrote: post #9

As millions of people interact with ChatGPT, their writing will subtly, gradually, begin to mimic its style. As future versions of the model are trained on this new text, both human and AI styles will converge until any difference between the two are infinitesimal.

One of the big complaints with LLMs is the confident hallucination of incorrect facts, like software APIs that don’t exist. But the way I see it, if ChatGPT thinks the Python list object should have a .is_sorted() property, that’s a pretty good indication that maybe it should. I work in PM (giant company, not Python), and one of these days my self-control will fail me and I will open a bug for “product does not suppo…

[deleted]
Post reply on HN