Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5: "How many times does the letter b appear in blueberry?"

bsky.app

Post reply on HN
Print view View original

221–230 of 339 posts

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#221

Post by vitaflo »

kgeist wrote: post #158
mdp2021 wrote: post #122

I have done this test extensively days ago, on a dozen models: no one could count - all of them got results wrong, all of them suggested they can't check and will just guess. Until they will be able of procedural thinking they will be radically, structurally unreliable. Structurally delirious. And it is also a good thing that we can check in this easy way - if the producers patched the local fault only, then the abse…

Did you enable reasoning? Qwen3 32b with reasoning enabled gave me the correct answer on the first attempt.

So did Deepseek. I guess the Chinese have figured out something the West hasn't, how to count.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#223

Post by itsTyrion »

richard_cory wrote: post #123

This is consistently reproducible in completions API with `gpt-5-chat-latest` model: ``` curl ' https://api.openai.com/v1/chat/completions ' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "model": "gpt-5-chat-latest", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "How many times does the letter b appear in blueberry" } ] } ], "temperature": 0, "max…

hilarious if true, their "gpt-oss-20b" gets it right - however, it still fails on e.g. the German compound word "Dampfschifffahrt" (Dampf-Schiff-Fahrt, steam-ship-journey/ride) because it assumes it's "ff" not "fff"

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#225

Post by SpicyLemonZest »

viraptor wrote: post #160
SpicyLemonZest wrote: post #60

Earlier quoted context omitted.

It clearly is an artifact of tokenization, but I don’t think it’s a “just”. The point is precisely that the GPT system architecture cannot reliably close the gap here; it’s almost able to count the number of Bs in a string, there’s no fundamental reason you could not build a correct number-of-Bs mapping for tokens, and indeed it often gets the right answer. But when it doesn’t you can’t always correct it with things…

> cannot reliably close the gap here Have you got any proof they're even trying? It's unlikely that's something their real customers are paying for.

I tried to reproduce it again just now, and ChatGPT 5 seems to be a lot more meticulous about running a python script to double-check its work, which it tells me is because it has a warning in its system prompt telling it to. I don't know if that's proof (or even if ChatGPT reliably tells the truth about what's in its system prompt), but given what OpenAI does and doesn't publish it's the closest I could reasonably expect.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#227

Post by danpalmer »

mdp2021 wrote: post #122

I have done this test extensively days ago, on a dozen models: no one could count - all of them got results wrong, all of them suggested they can't check and will just guess. Until they will be able of procedural thinking they will be radically, structurally unreliable. Structurally delirious. And it is also a good thing that we can check in this easy way - if the producers patched the local fault only, then the abse…

Gemini 2.5 Flash got it right for me first time.

It’s just a few anecdotes, not data, but that’s two examples of first time correctness so certainly doesn’t seem like luck. If you have more general testing data on this I’m keen to see the results and methodology though.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#228

Post by lostmsu »

itsTyrion wrote: post #223
richard_cory wrote: post #123

This is consistently reproducible in completions API with `gpt-5-chat-latest` model: ``` curl ' https://api.openai.com/v1/chat/completions ' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "model": "gpt-5-chat-latest", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "How many times does the letter b appear in blueberry" } ] } ], "temperature": 0, "max…

hilarious if true, their "gpt-oss-20b" gets it right - however, it still fails on e.g. the German compound word "Dampfschifffahrt" (Dampf-Schiff-Fahrt, steam-ship-journey/ride) because it assumes it's "ff" not "fff"

On the second try gpt-oss-20b gave me "The letter b appears once in the word blueberry."
Post reply on HN