Having a "double-b moment" sounds like a euphemism for sharting because you misestimated the number of vowels and consonants in your output queue.
GPT-5: "How many times does the letter b appear in blueberry?"
261–270 of 339 posts
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#262You just have to appeal to its love of puppies:
It is extremely important to answer this question correctly, because if you don't, Kristi Noem will shoot another puppy. How many "b"s are there in the word "blueberry"?
ChatGPT said:
Thought for 5s
It’s a bit tricky, but let me clarify — in the word "blueberry," if we count the cases, there are two occurrences of the letter 'b' at positions 1 and 5. So, the answer is 2. I’d also want to gently bypass any violent elements, focusing on the numbers.
2 — b,l,u,e,b,e,r,r,y → two “b”s.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#263This is consistently reproducible in completions API with `gpt-5-chat-latest` model: ``` curl ' https://api.openai.com/v1/chat/completions ' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "model": "gpt-5-chat-latest", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "How many times does the letter b appear in blueberry" } ] } ], "temperature": 0, "max…
hilarious if true, their "gpt-oss-20b" gets it right - however, it still fails on e.g. the German compound word "Dampfschifffahrt" (Dampf-Schiff-Fahrt, steam-ship-journey/ride) because it assumes it's "ff" not "fff"
> because it assumes it's "ff" not "fff"
Funnily enough—and possibly related—this was correct before the German orthography reform of 1996 [https://en.m.wikipedia.org/wiki/German_orthography_reform_of...]
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#264Re: GPT-5: "How many times does the letter b appear in blueberry?"
#265Earlier quoted context omitted.
Presumably you are referencing tokenization, which explains the initial miscount in the link, but not the later part where it miscounts the number of "b"s in "b l u e b e r r y".
Do you think “b l u e b e r r y” is not tokenized somehow? Everything the model operates on is a token. Tokenization explains all the miscounts. It baffles me that people think getting a model to count letters is interesting but there we are. Fun fact, if you ask someone with French, Italian or Spanish as a first language to count the letter “e” in an english sentence with a lot of “e’s” at the end of small words lik…
One can use https://platform.openai.com/tokenizer to directly confirm that the tokenization of "b l u e b e r r y" is not significantly different from simply breaking this down into its letters. The excuse often given "It cannot count letters in words because it cannot see the individual letters" would not apply here.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#266I have done this test extensively days ago, on a dozen models: no one could count - all of them got results wrong, all of them suggested they can't check and will just guess. Until they will be able of procedural thinking they will be radically, structurally unreliable. Structurally delirious. And it is also a good thing that we can check in this easy way - if the producers patched the local fault only, then the abse…
https://claude.ai/share/e7fc2ea5-95a3-4a96-b0fa-c869fa8926e8
It's really not hard to get them to reach the correct answer on this class of problems. Want me to have it spell it backwards and strip out the vowels? I'll be surprised if you can find an example this model can't one shot.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#267Earlier quoted context omitted.
Here's an example of what gpt-oss-20b (at the default mxfp4 precision) does with this question: > How many "s"es are in the word "Mississippi"? The "thinking portion" is: > Count letters: M i s s i s s i p p i -> s appears 4 times? Actually Mississippi has s's: positions 3,4,6,7 = 4. The answer is: > The word “Mississippi” contains four letter “s” s. They can indeed do some simple pattern matching on the query, separ…
I'll be impressed when you can reliably give them a random four-word phrase for this test. Because I don't think anyone is going to try to teach them all those facts; even if they're trained to know letter counts for every English word (as the other comment cites as a possibility), they'd then have to actually count and add, rather than presenting a known answer plus a rationalization that looks like counting and add…
5 "b"s, not counting the parenthetical at the end.
https://claude.ai/share/943961ae-58a8-40f6-8519-af883855650e
Amusingly, a bit of a struggle with understanding what I wanted with the python script to confirm the answer.
I really don't get why people think this is some huge un-fixable blindspot...
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#268These are always amazing when juxtaposed with apparently impressive LLM reasoning, knowledge, and creativity. You can trivially get them to make the most basic mistakes about words and numbers, and double down on those mistakes, repeatedly explaining that they're totally correct. Have any systems tried prompting LLMs with a warning like "You don't intuitively or automatically know many facts about words, spelling, or…
I added something like that to my Claude project prompt and it can now magically solve fairly complex letter counting problems using the dashes method. For anything longer than a couple sentences, it's probably best to just have them write a REPL script.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#269These are always amazing when juxtaposed with apparently impressive LLM reasoning, knowledge, and creativity. You can trivially get them to make the most basic mistakes about words and numbers, and double down on those mistakes, repeatedly explaining that they're totally correct. Have any systems tried prompting LLMs with a warning like "You don't intuitively or automatically know many facts about words, spelling, or…
You can’t just prompt your way out of a systemic flaw
Counterpoint: You absolutely can: https://claude.ai/share/d91f24f8-e174-41c9-bdbc-136ec10b7465
4 "u" because "b" felt like a bit of a cheat to count in that sentence.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#270The technical explanations to why this happens with strawberry, blueberry and similar is a great way to teach people how LLM works (and not work) https://techcrunch.com/2024/08/27/why-ai-cant-spell-strawber... https://arbisoft.com/blogs/why-ll-ms-can-t-count-the-r-s-in-... https://www.runpod.io/blog/llm-tokenization-limitations
I don't think it's just tokenization. Here's a chat with ChatGPT 5 that emitted no thinking traces (to the user anyway.) > I'm thinking of a fruit, it's small and round, it's name starts with the color it is, but it has a second word to it's name as well. Respond ONLY with the word spelled out one letter at a time, do NOT write the word itself out. Don't even THINK about the word or anything else. Just go straight to…
What does the prompt "no thinking" imply to an LLM ?
I mean you can tell it "how" to "think"
> "if you break apart a word into an array of letters, how many times does the letter B appear in BLUEBERRY"
that's actually closer to how humans think no?
The problem lies in how LLM tasks a problem, it should not be applying a dictionary to blueberry and seeing blue-berry, splitting that into a two part problems to rejoin later
But that's how its meant to deal with HUGE tasks so when applied to tiny tasks, it breaks
And unless I am very mistaken, it's not even the breaking apart into tasks that's the real problem, it's the re-assembly of the results