Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5: "How many times does the letter b appear in blueberry?"

bsky.app

Post reply on HN
Print view View original

241–250 of 339 posts

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#241

Post by mdp2021 »

skeledrew wrote: post #216
mdp2021 wrote: post #212

Earlier quoted context omitted.

If your prototype of the «knife» is all blade and no handle, fix it and implement the handle. If the creation is planned, you will have also thought of the handle; if it is a serendipity, you will have to plan the handle afterwards.

Pretty sure it doesn't matter to the child whether the knife has a handle or not. They'll eventually find a way to cut themself.

It matters to the adult - who is also an user.

LLMs do not deliver (they miss important qualities related to intelligence); they are here now; so they must be superseded.

There is no excuse: they must be fixed urgently.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#242

Post by atoav »

I had GPT-5 fail at the task of generating 5 songtitles with varying length. Then when I asked it how it obeyed my rule about varying length it tried to gaslight me by explaining these songtitles are 5 words long [lists 3-word songtitles].

Quite frankly, it feels these models got faster, but way worse.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#243

Post by mdp2021 »

kgeist wrote: post #213
mdp2021 wrote: post #204

Earlier quoted context omitted.

> they don't see individual letters Yet they seem to be from many other tests (characters corrections or manipulation in texts, for example). > The fact that reasoning models can count letters, even though they can't see individual letters To a mind, every idea is a representation. But we want the processor to work reliably on them representations. > If we don't allow a [mind] to base its reasoning on the training da…

Most people act on gut instincts first as well. Gut instinct = first semi-random sample from experience (= training data). That's where all the logical fallacies come from. Things like the bat and the ball problem, where 95% people give an incorrect answer, because most of the time, people simply pattern-match too. It saves energy and works well 95% time. Just like reasoning LLMs, they can get to a correct answer if…

> Most people act on gut instincts first as well

And we do not hire «most people» as consultants intentionally. We want to ask those intellectually diligent and talented.

> language model that is based on human representations

The machine is made to process the input - not to "intake" it. To create a mocker of average-joe would be an anti-service in both that * the project was to build a processor and * we refrain to ask average-joe. The plan can never have meant to be what you described, the mockery of mediocrity.

> we want better reliability

We want the implementation of a well performing mind - of intelligence. What you described is the "incompetent mind", the habitual fool - the «human mind model» is prescriptive based on what the properly used mind can do, not descriptive on what sloppy weak minds do.

> Can a "lesser" mind create a "greater" mind

Nothing says it could not.

> one free of human limitations

Very certainly yes, we can build things with more time, more energy, more efficiency, more robustness etc. than humans.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#244

Post by mdp2021 »

vitaflo wrote: post #221
kgeist wrote: post #158

Earlier quoted context omitted.

Did you enable reasoning? Qwen3 32b with reasoning enabled gave me the correct answer on the first attempt.

So did Deepseek. I guess the Chinese have figured out something the West hasn't, how to count.

No, DeepSeek also fails. (It worked in your test - it failed in similar others.)

(And note that DeepSeek can be very dumb - in practice, as experienced in our practice, and in standard tests, where it shows an ~80 IQ, where with other tools we achieved ~120 IQ (trackingai.org). DeepSeek was in important step, a demonstration of potential for efficiency, a gift - but it is still part of the collective work in progress.)

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#245

Post by skeledrew »

mdp2021 wrote: post #241
skeledrew wrote: post #216

Earlier quoted context omitted.

Pretty sure it doesn't matter to the child whether the knife has a handle or not. They'll eventually find a way to cut themself.

It matters to the adult - who is also an user. LLMs do not deliver (they miss important qualities related to intelligence); they are here now; so they must be superseded. There is no excuse: they must be fixed urgently.

LLMs deliver pretty well on their intended functionality: they predict next tokens given a token history and patterns in their training data. If you want to describe that as fully intelligent, that's your call, but I personally wouldn't. And adding functionality that isn't directly related to improving token prediction is just bad practice in an already very complex creation. LLM tools exist for that reason: they're the handles, sheaths, sharpeners, etc for the knife. Teach those adults who're getting themselves cut to hold the knife by the handle and use the other accessories that improve user experience.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#246

Post by acters »

andai wrote: post #18

My phone still has gpt-4o which gets it right: https://files.catbox.moe/0yg6cu.jpg But my browser has gpt-5 which says 3: https://files.catbox.moe/63qkce.jpg Claude spells it out letter by letter: https://files.catbox.moe/f1irfx.jpg So I thought GPT-5 Thinking might get it right, and it does: https://files.catbox.moe/xlchnr.jpg It refuses to show the thinking process for this question though, so its unclear if it eve…

I asked GPT 5 to spell out the individual letters of strawberry or blueberry. It did it correctly by essentially putting a space char in between the letters.

Then I simply asked it to count all unique letters in the word. GPT 5 still got it completely correct without thinking.

Lastly I asked how many r(or b) is in the word. This one for some reason switched to GPT 5 thinking with few seconds of reasoning. It out put the correct number.

I guess starting the conversation by painstakingly walking it over to the correct answer helps it out. Idk it's a silly test

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#247

Post by orwin »

flowerthoughts wrote: post #239
mikehearn wrote: post #132

This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.

No need to anthropomorphize. This is a tool designed for language understanding, that is failing at basic language understanding. Counting wrong might be bad, but this seems like a much deeper issue.

Transformers vectorize words in n dimensions before processing them, that's why they're very good at translation (basically they vectorize the English sentence, then devectorize in Spanish or whatever). Once the sentence is processed, 'blueberry' is a vector that occupy basically the same place as other berries, and probably other. The GPT will make a probabilistic choice (probably artificially weighted towards strawberry),and it isn't always blueberry.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#249

Post by nudgeOrnurture »

windowshopping wrote: post #144
mikehearn wrote: post #132

This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.

What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…

[flagged]

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#250

Post by mike_d »

spidersouris wrote: post #240
energy123 wrote: post #236

Earlier quoted context omitted.

The real criticism should be the AI doesn't say "I don't know.", or even better, "I can't answer this directly because my tokenizer... But here's a python snippet that calculates this ...", so exhibiting both self-awareness of limitations combined with what an intelligent person would do absent that information. We do seem to be an architectural/methodological breakthrough away from this kind of self-awareness.

For the AI to say this or to produce the correct answer would be easily achievable with post-training. That's what was done for the strawberry problem. But it's just telling the model what to reply/what tools to use in that exact situation. There's nothing about "self-awareness".

> But it's just telling the model what to reply/what tools to use in that exact situation.

So the exact same way we train human children to solve problems.

Post reply on HN