Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you build a ChatGPT detector?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Ask HN: How would you build a ChatGPT detector?

#91

Post by dragonwriter »

scotty79 wrote: post #90
abrichr wrote: post #47

Earlier quoted context omitted.

> Answer the question, "How would you build a ChatGPT detector?" written in the style of an optimistic Hacker News commenter Building a ChatGPT detector is a great idea, and I think it's something that could be really useful for a lot of people. Here's how I would approach it: First, I would start by training a large language model, like GPT-3, on a massive dataset of chat logs. This would give the model a good under…

> Overall, I think building a ChatGPT detector is a great idea, and I'm confident that with the right approach, it could be a valuable tool for anyone who uses chat platforms. I think that hollowly summing up and reiterating a point of the whole text in the last sentence might be a good signal to differentiate Open AI and humans. AI seems to be doing that on nearly all creative responses.

It ia a fairly widely taught grade school technique that many people never get beyond that is probably heavily represented in its training corpus.

Re: Ask HN: How would you build a ChatGPT detector?

#92

Post by pyuser583 »

qayxc wrote: post #4

I agree with the softwaredoug that this is probably not a technical problem. Any proposed solution would only serve to make the next iteration of the model better at avoiding detection (e.g. incorporating a GAN-style training cycle). Just as with Poe's law, there simply comes a point at which it becomes impossible to recognise AI generated texts just by analysing them. Made-up references might be a clue, but that's v…

“Poe’s law?” Is that Edgar Allen Poe?

Re: Ask HN: How would you build a ChatGPT detector?

#93

Post by NackerHughes »

jefftk wrote: post #55
zztop44 wrote: post #49

Earlier quoted context omitted.

This (human) ChatGPT detector says this text was written by a human.

I'm guessing bot, since "It's just a chatbot with a pre-determined set of responses" is super wrong

Wrong, but hilariously convincing as a hackernews-commenter-style post. If someone was given a load of hackernews comments and asked to find the one that was generated by a bot, who would look twice at that comment?

Re: Ask HN: How would you build a ChatGPT detector?

#94

Post by wslh »

Very basic: asking math questions: https://news.ycombinator.com/item?id=33840405

It seems people is blindly following ChatGPT like a new religion. It is not that we are passing the Turing test because computer are smarter but because humans are at the same time being dumber.

Re: Ask HN: How would you build a ChatGPT detector?

#95

Post by qayxc »

pyuser583 wrote: post #92
qayxc wrote: post #4

I agree with the softwaredoug that this is probably not a technical problem. Any proposed solution would only serve to make the next iteration of the model better at avoiding detection (e.g. incorporating a GAN-style training cycle). Just as with Poe's law, there simply comes a point at which it becomes impossible to recognise AI generated texts just by analysing them. Made-up references might be a clue, but that's v…

“Poe’s law?” Is that Edgar Allen Poe?

https://en.wikipedia.org/wiki/Poe%27s_law

Re: Ask HN: How would you build a ChatGPT detector?

#96

Post by constantcrying »

background wrote: post #9
qayxc wrote: post #4

I agree with the softwaredoug that this is probably not a technical problem. Any proposed solution would only serve to make the next iteration of the model better at avoiding detection (e.g. incorporating a GAN-style training cycle). Just as with Poe's law, there simply comes a point at which it becomes impossible to recognise AI generated texts just by analysing them. Made-up references might be a clue, but that's v…

Right, but in principle the detectors could iterate and get better over time too. That's why I asked about "a ChatGPT detector" instead of "general AI detector", which is a very different problem.

>but in principle the detectors could iterate and get better over time too

At some point the text will become undetectable. The same happens for image generating Networks. The detection works by detecting a statistical discrepancy between the AI output and "real" data. The goal of the AI in training is also to close that gap, the better the network, the harder it is to detect and the more output data you would need to get any significant results.

Re: Ask HN: How would you build a ChatGPT detector?

#97

Post by jokethrowaway »

xdennis wrote: post #18

According to ChatGPT: > 1. Start by training a machine learning model to identify features of natural language. You can use supervised learning techniques with a corpus of labeled data consisting of texts written by both humans and AI systems. > 2. Use natural language processing techniques to extract features from the text such as word frequencies, grammar, and syntax. > 3. Implement a scoring system to assign a num…

> For example, a frequent feature I've observed (of course, I'm not alone) is that it tends to generate lists of things and tie them together with "Firstly, item1. Additionally, item2. More so, item3. In conclusion, item4".

That's exactly how I approach anything and I'm 80% sure I'm not an AI (at least, not so unsophisticated)

Re: Ask HN: How would you build a ChatGPT detector?

#98

Post by redeyedtreefrog »

Right now ChatGPT responses tend to be very confidently asserted, are overly verbose, and contain factual errors. So right now you could build model(s) that measure the assertiveness and wordiness of text. But then people could just instruct ChatGPT to give brief answers, or to make itself sound uncertain. And soon there will be many similar models, and many will be more terse and less assertive by default. Regarding factual mistakes, that's not something you can check with a model that isn't itself more advanced than the LLM it is assessing.

Re: Ask HN: How would you build a ChatGPT detector?

#99

Post by sdwr »

sshine wrote: post #61
jefftk wrote: post #55

Earlier quoted context omitted.

I'm guessing bot, since "It's just a chatbot with a pre-determined set of responses" is super wrong

Regardless, it does sound like an pessimistic hacker news commenter. In other news: People discover interactive mirrors, and get freaked out!

Woof woof woof!

Re: Ask HN: How would you build a ChatGPT detector?

#100

Post by skilled »

lukeck wrote: post #70
skilled wrote: post #21

I'm not sure if you're only asking about specifics or you have a broader concern, but even if you build such a model, it wouldn't make any difference for cases where some kind of "gain" is involved. Be it financial, educational, etc. We're looking at a model that is able to parse the Web (but likely not all of it) and perform human-like actions on that information, and soon enough we'll have it work just like Stable…

If you ask it, it says it can’t search the web and only works with the data it knows about. You can already ask it to provide responses “in the style of.”

When I said parse, I meant it can take the data it already has.

It is still very average for "in the style of" prompts.

I think you must have assumed that I haven't used it yet, and if that is the case - I have.

Post reply on HN