GPT-5: "How many times does the letter b appear in blueberry?" • Hacker News Board

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#21

Post by simianwords » Fri, Aug 08, 2025, 3:45 PM UTC

fathermarz wrote: post #20

simianwords wrote: post #9

If you choose the thinking model it doesn’t make this mistake. It means the auto router should be tuned to call the thinking model on edge cases like these.

If you are going to release a new set of models and eliminate the old ones, your new smallest model should have equal capabilities equal to or greater than your old models of similar size. This is not a routing problem. This is a driving the price down of running and scaling the business problem.

do you not think 4o has the same capabilities as GPT-5 without thinking?

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#22

Post by amai » Fri, Aug 08, 2025, 6:29 PM UTC

Isn't that just an artifact caused by the tokenization of the training and input data?

See

https://platform.openai.com/tokenizer

https://github.com/openai/tiktoken

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#23

Post by dsadfjasdf » Fri, Aug 08, 2025, 6:34 PM UTC

philipwhiuk wrote: post #13

mikestorrent wrote: post #5

Earlier quoted context omitted.

This is a great idea. Like, if someone asked me to count the number of B's in your paragraph, I'd yeet it through `grep -o 'B' file.txt | wc -l` or similar, why would I sit there counting it by hand? As a human, if you give me a number on screen like 100000000, I can't be totally sure if that's 100 Million or 1 Billion without getting close and counting carefully. Should ought have my glasses. Mouse pointer helps som…

If you have to build an MCP for every system you aren’t building intelligence in the first place.

Why? just cause? analogize it to the human brain.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#24

Post by dang » Sat, Aug 09, 2025, 9:46 PM UTC

Url changed from https://bsky.app/profile/kjhealy.co/post/3lvtxbtexg226, which points to this.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#25

Post by falcor84 » Sat, Aug 09, 2025, 9:52 PM UTC

amai wrote: post #22

Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://github.com/openai/tiktoken

Where in the tokenization does the 3rd b come from?

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#26

Post by andrewmcwatters » Sat, Aug 09, 2025, 9:53 PM UTC

amai wrote: post #22

Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://github.com/openai/tiktoken

No, it's the entire architecture of the model. There's no real reasoning. It seems that reasoning is just a feedback loop on top of existing autocompletion.

It's really disingenuous for the industry to call warming tokens for output, "reasoning," as if some autocomplete before more autocomplete is all we needed to solve the issue of consciousness.

Edit: Letter frequency apparently has just become another scripted output, like doing arithmetic. LLMs don't have the ability to do this sort of work inherently, so they're trained to offload the task.

Edit: This comment appears to be wildly upvoted and downvoted. If you have anything to add besides reactionary voting, please contribute to the discussion.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#27

Post by wslh » Sat, Aug 09, 2025, 9:55 PM UTC

Seems like they just fixed it: [1]. A "thinking longer for a better answer" message appeared before giving the answer.

[1] https://chatgpt.com/share/6897c38b-12b8-800d-9cc2-571adb13bc...

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#28

Post by lottin » Sat, Aug 09, 2025, 9:57 PM UTC

skeledrew wrote: post #19

philipwhiuk wrote: post #12

Earlier quoted context omitted.

You can’t just prompt your way out of a systemic flaw

What's the systematic flaw?

The fact that it can't count.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#29

Post by [deleted] » Sat, Aug 09, 2025, 9:58 PM UTC

wslh wrote: post #27

Seems like they just fixed it: [1]. A "thinking longer for a better answer" message appeared before giving the answer. [1] https://chatgpt.com/share/6897c38b-12b8-800d-9cc2-571adb13bc...

[deleted]