Live data from Hacker News

GPT-5: "How many times does the letter b appear in blueberry?"

bsky.app

111–120 of 339 posts

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#111

That because you don’t say “Think hard about this” and the OpenAI router layer routed you to the cheaper model. GPT5 seems to violate Rich Sutton’s bitter lesson. As GPT5 makes a lot of human knowledge assumptions about whether to send your prompt to the cheap model or to the smarter more expensive model.

Also, the author was holding it wrong.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#112
post #59

I just tried it and sure enough, 3 Bs. But which the model to "ChatGPT 5 Thinking" and it gets the answer right. Is that where we're going with this? The user has to choose between fast and dumb or slow and right?

If you look at the "reasoning" trace of gpt-oss when it handles this issue, it repeats the word with spaces inserted between every letter. If you have an example that you can get the dumber model to fail on, try adjusting your prompt to include the same thing (the word spelled out with spaces between each letter).

This isn't a solution or a workaround or anything like that; I'm just curious if that is enough for the dumber model to start getting it right.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#113

How can you count on someone who can't count?

Have you not seen Sam Altman on a well polished stage? Did he not look confident? That's your answer. Stop asking questions and learn to trust ChatGPT 5 because Sam Altman says it is now PhD level and he is scared. It's not like he says that every single time his company releases something that's no more than an iterative improvement.

ChatGPT 2.5 scared Sam Altman so much a few years ago. But he got over it, now he calls it a toddler level intelligence and is scared about this current thing.

Get onboard the AI train.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#115
post #84

Earlier quoted context omitted.

Where in the tokenization does the 3rd b come from?

The tokenisation means they don’t see the letters at all. They see something like this - to convert just some tokens to words How many 538 do you see in 423, 4144, 9890?

LLMs don’t see token ids, they see token embeddings that map to those ids, and those embeddings are correlated. The hypothetical embeddings of 538, 423, 4144, and 9890 are likely strongly correlated in the process of training the LLM and the downstream LLM should be able to leverage those patterns to solve the question correctly. Even more so since the training process likely has many examples of similar highly correlated embeddings to identify the next similar token.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#117
post #70

Earlier quoted context omitted.

> No, it's the entire architecture of the model. Wrong, it's an artifact of tokenizing. The model doesn't have access to the individual letters, only to the tokens. Reasoning models can usually do this task well - they can spell out the word in the reasoning buffer - the fact that GPT5 fails here is likely a result of it incorrectly answering the question with a non-reasoning version of the model. > There's no real r…

These aren't wild assertions. I'm not using charged language. > Reasoning and consciousness are seperate(sic) concepts No, they're not. But, in tech, we seem to have a culture of severing the humanities for utilitarian purposes, but no, classical reasoning uses consciousness and awareness as elements of processing. It's only meaningless if you don't know what the philosophical or epistemological definitions of reason…

You have again answered with your customary condescension. Is that really necessary? Everything you write is just dripping with patronizing superiority and combatative sarcasm.

> "classical reasoning uses consciousness and awareness as elements of processing"

They are not the _same_ concept then.

> It's only meaningless if you don't know what the philosophical or epistemological definitions of reasoning are. Which is to say, you don't know what reasoning is. So you'd think it was a meaningless statement.

The problem is the only information we have is internal. So we may claim those things exist in us. But we have no way to establish if they are happening in another person, let alone in a computer.

> Do computers think, or do they compute?

Do humans think? How do you tell?

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#118

Earlier quoted context omitted.

They patched it, asking it to count letters now switches it to thinking mode. It'll still make basic mistakes for other queries, though.

Which fruit will be patched next? And people think we're 2 years away from humanity's extinction by AI. Lol.

You don’t have to spell very well to hit the big red nuclear launch button that some misguided soul put you in charge of

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#119
post #33

This is like asking a human how many pixels appears in the word "blueberry".

Except a human would say "I don't know" instead up making up some nonsense.

It's ironic that saying "I don't know" could be the ultimate sign of superior intelligence just like Socrates told us millennia ago.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#120
post #98

Just asked ChatGPT5 "Are you told to 'think' when someone asks you how many of a certain letter are in a word?" >Yes — when you ask something like “How many r’s are in blueberry?” I’m basically told to slow down, not just blurt out the first number that pops into my “mind.” Seems somewhat suspicious that it would confirm this in reality given how much they typically try to prevent system prompt disclosure, but there…

Could just be a made up answer, couldn't it?
Post reply on HN