Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTZero Case Study – Exploring False Positives

gonzoknows.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: GPTZero Case Study – Exploring False Positives

#111

Post by Jensson »

valine wrote: post #107
Jensson wrote: post #101

Earlier quoted context omitted.

If significantly bigger models than now got better results we would have seen papers about that a long time ago so that the team/company can get more funding, lots of rich actors has worked on that for years. If it doesn't produce better results however then they want their competitors to waste lots of money to make the same mistakes, there is really no benefit from publishing that and lots of drawbacks. Otherwise it…

It’s not just the cost of training the model, it’s the cost of doing inference at scale. ChatGPT boarder line too expensive to operate already. It’s hard to imagine a larger model that both economical and used by millions of people with our current hardware.

But if a larger model was good enough to replace a human, like for example a Google engineer, then it would still be worth it. So they have for sure tried to scale up, and if that extra scale gave results they would have published something about it.

Now, since the larger model wasn't good enough to replace a human engineer we can rest easy, it wont replace programmers anytime soon. If GPT-4 for example could replace engineers, OpenAI wouldn't need to monetise ChatGPT, they would just rent out artificial engineers to do coding for $10k a year.

Re: GPTZero Case Study – Exploring False Positives

#112

Post by d0mine »

brookst wrote: post #8

I’ve got a fun little side project that uses GPT. I tested gptzero against 10 of my projects’ writings and 10 of my own. It detected 6 out of 10 correctly in both cases (4 gpt-written bits were declared human, 4 human-written were declared gpt). Which is better than 50% but not nearly good enough to base any kind of decision on.

> which is better than 50%

Unrelated: p-value for getting 12 from 20 correct just by chance is ~0.4 that is there is not enough data for the conclusion "better" in this case.

Null hypothesis: 50%/50%, the result random, normal distribution:

  H0: p=1/2
  H1: p!=1/2 (two-tail) 


  import statistics
  
  p0 = 0.5  # proportion of successes according to null hypothesis
  n = 20   # sample size
  p_sample = 12/n  # 12 from 20 are correct
  
  sigma = (p0 * (1 - p0) / n)**.5  # std according to H0
  z_score = (p_sample - p0) / sigma  # test statistic
  p_value = 2*statistics.NormalDist().cdf(-abs(z_score))  # prob. two-tails
  # p-value -> 0.4

Re: GPTZero Case Study – Exploring False Positives

#113

Post by e2le »

wand3r wrote: post #12

I saw this[1] interview with Sam Altman touching on interim AI impact. I really agree with his point that basically detecting output from LLMs is basically going to be futile and only really relevant in the near term. Accuracy is obviously going to improve in models and detection isnt that difficult now but will be in the future, especially if output is modified or an attempt to obfuscate origin is made. [1] https://…

Perhaps instead, the solution is to hold people to a higher standard. AI as a tool to help people in identifying logical fallacies (special pleading, anecdotal, the fallacy fallacy) written or otherwise could be very valuable. Maybe in the same way that ChatGPT can identify and explain the problems a piece of code has. It could even be integrated with moderation tools. This could in-effect make individuals less vulnerable to convincing nonsense said by professional bullshitters or parroting individuals.

Re: GPTZero Case Study – Exploring False Positives

#114

Post by sebzim4500 »

Closi wrote: post #85
theptip wrote: post #75

Earlier quoted context omitted.

Of course false positives matter; the naive heuristic “everything is AI generated” has zero false negatives, and mostly false positives. In the OP half of the positives are false. That’s not a useful signal IMO. You couldn’t use that to police homework for example.

I said they might not matter for quite a few use cases , not that they don't matter for all use-cases. e.g. If you were Sam Altman at OpenAI and your use-case is mostly looking for training data and wanting to tell if it is AI-Generated or not (so you can exclude this from training data), you probably care much more about false negatives than false positives (false positives just reduce your training data set size sl…

Even for that the false positives could end up mattering. E.g. if the data being incorrectly excluded turns out to be the most important part of the training set.

Re: GPTZero Case Study – Exploring False Positives

#115

Post by air7 »

I read an interesting paper about an idea of watermarking LLM output text in such a way that makes detection very accurate for a long enough text. This is done by subtly changing the probabilities of the next word to be generated based on the last word that was outputed. Circumventing it by manually changing words post hoc would potentially require almost as much work as writing it from scratch.

The idea seems quite roboust to me and I can envisage a future where companies that provide access to LLMs would also publish a detection tool for their models.

Re: GPTZero Case Study – Exploring False Positives

#116

Post by cypress66 »

Closi wrote: post #52
sebzim4500 wrote: post #44

Earlier quoted context omitted.

>detection isnt that difficult now I would have thought this, but every attempt I've seen at detecting chatGPT generated text has failed miserably.

It fails on false positives, but you don't often get false negatives, which might be enough at the moment for quite a few use-cases. Also false positives are typically "this text is likely to contain parts that were AI generated" rather than "This text is higly likely to be AI generated" (which is what GPT-generated content generally produces). When I've tried to prompt-engineer GPT to produce text that GPTZero will…

I've read that you can literally explain chatgpt what perplexity and burstiness is, and tell it to generate it's answer with low perplexity and high burstiness (what detectors check).

I haven't tried it though.

Re: GPTZero Case Study – Exploring False Positives

#117

Post by brookst »

supriyo-biswas wrote: post #69
brookst wrote: post #10

Earlier quoted context omitted.

The best was the Ted Chiang article making numerous category errors and forest/trees mistakes in arguing that LLMs just store lossy copies of their training data. It was well-written, plausible, and so very incorrect.

Neural network based compression algorithms[1] are a thing, so I believe Ted Chiang's assessment is right. Memorization (albeit lossy) is also how the human brain works and develops reasoning[2]. [1] https://bellard.org/nncp/ [2] https://www.pearlleff.com/in-praise-of-memorization

The fact that some neural network architectures can compress data does not mean that data compression is the only thing any neural network can do.

It’s like saying that GPUs can render games, so GPT is a game because it uses GPU.

Re: GPTZero Case Study – Exploring False Positives

#118

Post by tluyben2 »

air7 wrote: post #115

I read an interesting paper about an idea of watermarking LLM output text in such a way that makes detection very accurate for a long enough text. This is done by subtly changing the probabilities of the next word to be generated based on the last word that was outputed. Circumventing it by manually changing words post hoc would potentially require almost as much work as writing it from scratch. The idea seems quite…

It's not hard to make a model that rewrites the text without changing the meaning which fails this. Our model[0] which is based on feeding chatgpt random things from the interwebs from before 2020 and letting it wobble on about it is pretty good and nice to play with, but it's pretty easy to change the score radically with just changing a few words. This is whack-a-mole no matter how it's done. For now you can be sure people are too lazy to do it, but there will be many tools in the future to evade tools like this.

[0] https://filteroutai.com/validate/a07e081b71b294ba2de236441be... https://filteroutai.com/validate/2c3fa6de32845df02be7a4ff185...

Re: GPTZero Case Study – Exploring False Positives

#119

Post by snapcaster »

RC_ITR wrote: post #106
valine wrote: post #60

Earlier quoted context omitted.

GPT3 is far more accurate than GPT2. Seems reasonable that larger models trained on more data will continue to improve accuracy. I'd also expect larger models to be better at summarizing text, ie potentially fixing the Bing issues where it hallucinates numbers. Our models sizes are a product of our scaling and hardware limitations. There's no reason to believe we are anywhere near optimal.

> GPT3 is far more accurate than GPT2 Please say more about what you mean here because I disagree. It’s certainly more eloquent, but it still can’t multiple 2 4-digit numbers…

But it did learn some basic arithmetic. If GPT4 can multiple 2 4-digit numbers will you change your mind?

Re: GPTZero Case Study – Exploring False Positives

#120

Post by RC_ITR »

snapcaster wrote: post #119
RC_ITR wrote: post #106

Earlier quoted context omitted.

> GPT3 is far more accurate than GPT2 Please say more about what you mean here because I disagree. It’s certainly more eloquent, but it still can’t multiple 2 4-digit numbers…

But it did learn some basic arithmetic. If GPT4 can multiple 2 4-digit numbers will you change your mind?

GPT saw a bunch of arithmetic and can repeat it. That’s the joke behind the Reddit usernames and /r/counting.

The four digit number thing is just the current lower bound of where it gets confused because a lack of training data.

Once you teach a patient 8 year old the rules of multiplication once, they can multiple any two numbers (that they’ve never seen before) with an arbitrary number of digits. An LLM cannot and will not ever be able to do that because it is a specific tool and it is not designed to do that (doing that would be a bad outcome for an LLM since we have different tools that can do multiplication much more efficiently).

So yes, if a LLM learns rules based math (which it is not intended to do) I’ll eat not only my, but every hat in existence.

Post reply on HN