Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you build a ChatGPT detector?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 109 posts

Re: Ask HN: How would you build a ChatGPT detector?

#11

Post by aiieeeeee »

It is certainly possible to build an AI that is capable of recognizing the writing style of another AI, such as ChatGPT. This kind of AI is often referred to as a "style transfer" or "style transfer model."

To build a ChatGPT detector, you would first need to collect a large amount of text that was generated by ChatGPT. This could be done by providing a variety of different prompts to the ChatGPT model and collecting the responses.

Next, you would need to train a machine learning model on this text. The most common approach for this kind of task is to use a deep learning model, such as a recurrent neural network (RNN). The RNN would be trained to take in a piece of text and predict whether it was written by ChatGPT or not.

As for the accuracy of such a model, it would depend on a variety of factors, including the quality and quantity of the training data, the specific model architecture and hyperparameters used, and the skill of the person training the model. With a large and diverse training dataset, and appropriate model design and training, it is likely that a ChatGPT detector could achieve relatively high accuracy.

Re: Ask HN: How would you build a ChatGPT detector?

#12

Post by baq »

background wrote: post #9
qayxc wrote: post #4

I agree with the softwaredoug that this is probably not a technical problem. Any proposed solution would only serve to make the next iteration of the model better at avoiding detection (e.g. incorporating a GAN-style training cycle). Just as with Poe's law, there simply comes a point at which it becomes impossible to recognise AI generated texts just by analysing them. Made-up references might be a clue, but that's v…

Right, but in principle the detectors could iterate and get better over time too. That's why I asked about "a ChatGPT detector" instead of "general AI detector", which is a very different problem.

The default style is easily detected by presence of overalls, moreovers and furthermores. When you tell it to do a ‘concise HN comment’ the only tell may be that its English is flawless. E.g.

In a style of a very concise HN comment describe how to detect that a text has been written by the Assistant.

To detect if a text has been written by the Assistant, check for the use of generic language, repetitive phrases, and a lack of personal perspective or opinion.

Re: Ask HN: How would you build a ChatGPT detector?

#15

Post by lordgroff »

aiieeeeee wrote: post #11

It is certainly possible to build an AI that is capable of recognizing the writing style of another AI, such as ChatGPT. This kind of AI is often referred to as a "style transfer" or "style transfer model." To build a ChatGPT detector, you would first need to collect a large amount of text that was generated by ChatGPT. This could be done by providing a variety of different prompts to the ChatGPT model and collecting…

I know this is fun for people but I have to say the GPT reply is getting old fast.

Re: Ask HN: How would you build a ChatGPT detector?

#16

Post by alkonaut »

aiieeeeee wrote: post #11

It is certainly possible to build an AI that is capable of recognizing the writing style of another AI, such as ChatGPT. This kind of AI is often referred to as a "style transfer" or "style transfer model." To build a ChatGPT detector, you would first need to collect a large amount of text that was generated by ChatGPT. This could be done by providing a variety of different prompts to the ChatGPT model and collecting…

Then I imagine you could use it to ensure the next version doesn’t score highly on the “probable bot” score. Sounds like an adversarial arms race.

Re: Ask HN: How would you build a ChatGPT detector?

#17

Post by devit »

Great question! I've been using the new AI assistant for a while now and I've definitely noticed a certain "sameness" to its responses. In fact, I think it would be relatively easy to build an AI that could recognize the writing of this AI. The key would be to train the AI on a large dataset of the assistant's responses, which would allow it to learn the patterns and characteristics of the assistant's writing style. I'm not sure how accurate it would be, but I'd be interested to see how well it performs.

Re: Ask HN: How would you build a ChatGPT detector?

#18

Post by xdennis »

According to ChatGPT:

> 1. Start by training a machine learning model to identify features of natural language. You can use supervised learning techniques with a corpus of labeled data consisting of texts written by both humans and AI systems.

> 2. Use natural language processing techniques to extract features from the text such as word frequencies, grammar, and syntax.

> 3. Implement a scoring system to assign a numerical score to each text based on the features identified by the model.

> 4. Create a threshold score which will be used to differentiate between texts written by humans and AI systems.

> 5. Test the model on a separate dataset to ensure accuracy.

> 6. Deploy the model in a production environment to detect AI-written texts.

This is from the prompt: "How would you build a tool to detect texts written by artificial intelligence systems?"

It seems like an okay plan, but doesn't offer any actual insight into differences between human text and AI text.

For example, a frequent feature I've observed (of course, I'm not alone) is that it tends to generate lists of things and tie them together with "Firstly, item1. Additionally, item2. More so, item3. In conclusion, item4".

Re: Ask HN: How would you build a ChatGPT detector?

#19

Post by baq »

lordgroff wrote: post #15
aiieeeeee wrote: post #11

It is certainly possible to build an AI that is capable of recognizing the writing style of another AI, such as ChatGPT. This kind of AI is often referred to as a "style transfer" or "style transfer model." To build a ChatGPT detector, you would first need to collect a large amount of text that was generated by ChatGPT. This could be done by providing a variety of different prompts to the ChatGPT model and collecting…

I know this is fun for people but I have to say the GPT reply is getting old fast.

The default style is such, arguably by design, exactly so it’s easily detected and gets downvoted on forums like here. IMHO it’s an important feature, even if it wasn’t intended for this purpose.

Re: Ask HN: How would you build a ChatGPT detector?

#20

Post by Jabrov »

devit wrote: post #17

Great question! I've been using the new AI assistant for a while now and I've definitely noticed a certain "sameness" to its responses. In fact, I think it would be relatively easy to build an AI that could recognize the writing of this AI. The key would be to train the AI on a large dataset of the assistant's responses, which would allow it to learn the patterns and characteristics of the assistant's writing style.…

This was written by a bot. I wish you would label it.
Post reply on HN