Maybe detecting grammar and spelling mistake will give you a first hint
How? People make these mistakes too
Ask HN: How would you build a ChatGPT detector?
81–90 of 109 posts
Re: Ask HN: How would you build a ChatGPT detector?
#82Given a prompt or partial output, human continuations follow some unknown probability distribution over the set of possible next words. Suppose that ChatGPT has perfectly learned this distribution - it can correctly tell you the probability of any possible next word. Does that make ChatGPT indistinguishable from human speech? Not necessarily, because there is also the question of how one samples actual outputs from ChatGPT's predicted distribution.
For example, suppose that OpenAI did the simplest thing, and always returned the most likely next word (in other words, sampling with temperature = 0). In that case, the perplexity of ChatGPT's outputs would be consistently lower than that of human speech, even according to a weaker language model.
In reality, ChatGPT's sampling is surely more subtle than this, but it is still unlikely to exactly match its predicted distribution - OpenAI wants to produce useful responses, so it will sample in a way that favors more probable continuations. I would expect that if you passed ChatGPT's outputs to a weaker language model and evaluated their likelihood, and compared it to the likelihood of a known-human corpus, it would quickly become apparent which is which.
After all, OpenAI's goal is for ChatGPT to sound human to a human reader (in addition to other goals like being helpful, being honest, not providing harmful information, etc.), not be statistically indistinguishable from a human to a language model.
Re: Ask HN: How would you build a ChatGPT detector?
#83Earlier quoted context omitted.
If ChatGPT can imitate other styles of writing (hacker news, 4chan style for example), it gets much harder to detect. I think within a few years it will become infeasible to say whether something was authored by an AI
It's already been prevented from emulating styles (at least directly): > As a large language model trained by OpenAI, I do not have any knowledge of specific styles or conventions used on Hacker News or any other online forum. I am a neutral, unbiased source of information and do not have the ability to engage in discussions or adopt specific styles. My purpose is to provide accurate and helpful information to the be…
It still works for me:
> Extol the virtues of your favorite note-taking software, in the style of a Hacker News comment.
I absolutely love using Evernote for taking notes. It's so versatile and user-friendly. The ability to tag and organize notes makes it easy to find exactly what I need, even if I have a ton of notes saved. Plus, the ability to add images, audio, and other attachments to my notes is a game-changer. And the fact that it syncs across all my devices makes it perfect for on-the-go note-taking. Evernote is definitely my go-to for all my note-taking needs.
Without “in the style of”:
My favorite note-taking software is Evernote. I love how user-friendly and intuitive it is to use, and how it allows me to easily organize and categorize my notes. I can easily add tags to my notes to make them easier to search and find, and I can even add audio and video notes to capture more information.
Evernote also offers a wide range of features that help me stay organized and productive. I can create to-do lists, set reminders, and even collaborate with other users on shared notes and notebooks. I can also access my notes on any device, including my phone and tablet, so I always have my notes with me wherever I go.
Another great thing about Evernote is its ability to integrate with other tools and services. I can easily save articles and webpages to Evernote, or even send emails directly to my Evernote account. This allows me to save and organize important information from a wide range of sources, all in one place.
Overall, I highly recommend Evernote to anyone looking for a powerful and versatile note-taking tool. It has truly changed the way I take and organize notes, and has helped me stay on top of my tasks and projects.
> Extol the virtues of your favorite note-taking software, in the style of 4chan.
Yo, if you're not using my favorite note-taking software, you're missing out, bro. This shit is like a god-tier tool for organizing your thoughts and keeping track of important information. It's got a clean, intuitive interface and a ton of features that make it super easy to use. Plus, it syncs across all your devices, so you can access your notes anywhere. It's like a fucking lifesaver for when you need to remember something important or jot down some ideas. Seriously, try it out and thank me later.
(By the way, did Evernote sponsor OpenAI?)
Re: Ask HN: How would you build a ChatGPT detector?
#84The solution likely involves having some verification process of humanhood and not the other way around.
Re: Ask HN: How would you build a ChatGPT detector?
#85[deleted]
Re: Ask HN: How would you build a ChatGPT detector?
#86[deleted]
Re: Ask HN: How would you build a ChatGPT detector?
#87Lol Maybe that’s openai business model charge teachers to see if a paper was written by them. Sadly jokes on them. Most teachers aren’t the money bags some think they are.
Re: Ask HN: How would you build a ChatGPT detector?
#882nd order Turing test: two AIs try to convince a human that the other AI is a robot.
Re: Ask HN: How would you build a ChatGPT detector?
#89I suspect the only solid solution is OpenAI themselves storing all text their models generates, and providing an API which will return whether they've outputted a specific string (or a similar one) in the past.
A lot of suggestions here talk about the consistent stylistic choices that ChatGPT makes, like it's lists or other particular mannerisms. I'd argue these are simply artefacts of it being fine-tuned on a large number of 'well-behaved' examples from Open AI. This phenomena is called partial mode collapse, this article does a great job discussing it with respect to GPT-3 [0].
Of course you could train a model to detect when this mode-collapse occurs to detect ChatGPT. The un-finetuned model, however, does not have these problems, so it's only a matter of OpenAI improving their fine-tuning dataset to return to an 'undetectable' AI.
[0] https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-...
Re: Ask HN: How would you build a ChatGPT detector?
#90> Answer the question, "How would you build a ChatGPT detector?" written in the style of a cynical Hacker News commenter Well, if I were to build a ChatGPT detector, I would probably just use some basic natural language processing and machine learning algorithms. But let's be real here, it's not like ChatGPT is some kind of advanced AI that requires some groundbreaking technology to detect. It's just a chatbot with a…
> Answer the question, "How would you build a ChatGPT detector?" written in the style of an optimistic Hacker News commenter Building a ChatGPT detector is a great idea, and I think it's something that could be really useful for a lot of people. Here's how I would approach it: First, I would start by training a large language model, like GPT-3, on a massive dataset of chat logs. This would give the model a good under…
> Overall, I think building a ChatGPT detector is a great idea, and I'm confident that with the right approach, it could be a valuable tool for anyone who uses chat platforms.
I think that hollowly summing up and reiterating a point of the whole text in the last sentence might be a good signal to differentiate Open AI and humans. AI seems to be doing that on nearly all creative responses.