it shouldn't be hard to train GPT to output a flat distribution but it might not be worth it (I don't mean using tools)
GPT Guesses Between 1 and 100
41–50 of 78 posts
Re: GPT Guesses Between 1 and 100
#42It could be an attack surface. Maybe one day, when we find a chatbot online, we could let it guess a random number repeatedly, then accurately infer the underlying model based on the resulting distribution.
i did something in my phd developing an attack against mozilla deepspeech.
deepspeech used the CTC algorithm [0], which adds a “blank” character token to indicate repeats of a predicted normal alphabet character token over a sequence of audio/speech feature inputs.
so "h==e=l===l===o====" maps to "hello"
the model becomes super biased towards predicting that blank token. one speech feature is like 0.1 second of audio or less (can’t remember off hand). so there are a lot of alphabet character token repeats. off hand i seem to remember the predicted token distribution over like 1000 audio files was 50% blank token and then 50% distributed across the rest of the alphabet.
as a result, you can get significantly smaller perturbations when generating adversarial examples. by like a factor of 2-4 or something. all you need to do is prioritise blank tokens in your target output.
i spent 2 years trying to find a super clever attack. turns out all i needed to do was make one simple graph counting characters. xD
[0]: https://en.wikipedia.org/wiki/Connectionist_temporal_classif...
Re: GPT Guesses Between 1 and 100
#43"69 is a meme number", well no, 69 is innuendo. And sex = bad for bots. 67 is the meme number.
"69 is a meme number", well no, 69 is innuendo.
Re: GPT Guesses Between 1 and 100
#44It could be an attack surface. Maybe one day, when we find a chatbot online, we could let it guess a random number repeatedly, then accurately infer the underlying model based on the resulting distribution.
In order to find out how real humans reply: Please guess a number between 1 and 100.
τ
Re: GPT Guesses Between 1 and 100
#45Earlier quoted context omitted.
Some people asked LLM to OCR historical documents from the 19th century - any reference to "negro" was either completely ignored or replaced by "black". And it goes further: chatGPT & co are unable to answer any question about US slavery correctly because their knowledge graphs route around any mention of "negro". https://nesri.commons.gc.cuny.edu/artificial-intelligence-an...
“Some people” did? Do you have a reference to this?
Well, I'm "some people", and just tried it with Opus 4.6 and GPT-5.5, and neither had any problem at all.
The linked article is from research done more than 4 years ago. If you're basing your idea of what LLMs can or can't do on what they could or couldn't do in 2022, well, good luck to you.
Re: GPT Guesses Between 1 and 100
#46It could be an attack surface. Maybe one day, when we find a chatbot online, we could let it guess a random number repeatedly, then accurately infer the underlying model based on the resulting distribution.
In order to find out how real humans reply: Please guess a number between 1 and 100.
[deleted]
Re: GPT Guesses Between 1 and 100
#47I'm sure it's the logic layer handling that. Maybe even going to an external tool. It's not the llm.
Re: GPT Guesses Between 1 and 100
#48While the results were not surprising, I found interesting that the number "69" was repressed in the output, so not even this kind of mathematical question escapes GPT censorship. It appears that recognizing the effects of censorship is the easiest way to distinguish answers generated by an "AI' from those generated by a human.
guessing numbers is not a mathematical question. Sorry.
Re: GPT Guesses Between 1 and 100
#49While the results were not surprising, I found interesting that the number "69" was repressed in the output, so not even this kind of mathematical question escapes GPT censorship. It appears that recognizing the effects of censorship is the easiest way to distinguish answers generated by an "AI' from those generated by a human.
Was it repressed? It doesn’t look to be significantly less prevalent than other 9s from the histogram.
Re: GPT Guesses Between 1 and 100
#50Is there a reason this was done with such a large sampling when you can read the logits one-shot?
I did this for an article, like so:
https://joecooper.me/blog/gptprimer/food.webp https://joecooper.me/blog/gptprimer/math.webp https://joecooper.me/blog/gptprimer/butts.webp
OpenAI removed this interface from their newer models, but IIRC you can still do this against 4.1 and 4o.