The defensive stance of some of the people in this thread is telling. The absolute meltdown that’s going to occur when humanity full internalizes the fact that LLMs are not and will never be intelligent is going to be of epic proportions.
GPT-5: "How many times does the letter b appear in blueberry?"
181–190 of 339 posts
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#182Earlier quoted context omitted.
What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…
Really? I thought the analogy was pretty good. Here senses refer to how the machines perceive text, IE as tokens that don't correspond 1:1 to letters. If you prefer a tighter comparison, suppose you ask an English speaker how many vowels are in the English transliteration of a passage of Chinese characters. You could probably figure it out, but it's not obvious, and not easy to do correctly without a few rounds of ca…
There is only 1 even number, Dave.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#183Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://github.com/openai/tiktoken
Common misconception. That just means the algorithm for counting letters can't be as simple as adding 1 for every token. The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights. If you're fine appealing to less concrete ideas, transformers are arbitrary function approximators, tokenization doesn't change that…
> The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights.
You seem to suppose that they actually perform addition internally, rather than simply having a model of the concept that humans sometimes do addition and use it to compute results. Why?
> For any finite-length function (like counting letters in a bounded domain), it's just a matter of having a big enough network and figuring out how to train it correctly. They just haven't bothered.
The problem is that the question space grows exponentially in the length of input. If you want a non-coincidentally-correct answer to "how many t's in 'correct horse battery staple'?" then you need to actually add up the per-token counts.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#184This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.
What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…
Analogies are just that, they are meant to put things in perspective. Obviously the LLM doesn't have "senses" in the human way, and it doesn't "see" words, but the point is that the LLM perceives (or whatever other word you want to use here that is less anthropomorphic) the word as a single indivisible thing (a token).
In more machine learning terms, it isn't trained to autocomplete answers based on individual letters in the prompt. What we see as the 9 letters "blueberry", it "sees" as an vector of weights.
> Illusions don't fool our intelligence, they fool our senses
That's exactly why this is a good analogy here. The blueberry question isn't fooling the LLMs intelligence either, it's fooling its ability to know what that "token" (vector of weights) is made out of.
A different analogy could be, imagine a being that had a sense that you "see" magnetic lines, and they showed you an object and asked you where the north pole was. You, not having this "sense", could try to guess based on past knowledge of said object, but it would just be a guess. You can't "see" those magnetic lines the way that being can.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#185Qwen3 32b with reasoning (which I run locally) gives the correct answer. A pretty good model for its size. Pretty sure GPT5 with reasoning should be able to solve it, too. I guess the real problem here is that GPT5's router doesn't understand that it's a problem which requires reasoning.
Can it count all 3 letter words in a paragraph? Can it average number of words inside parentheses in a paragraph?
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#186This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.
In an optical illusion, we perceive something that isn't there due to exploiting a correction mechanism that's meant to allow us to make better practical sense of visual information in the average case. Asking LLMs to count letters in a word fails because the needed information isn't part of their sensory data in the first place (to the extent that a program's I/O can be described as "sense"). They reason about text…
I can't take credit for coming up with this, but LLMs have basically inverted the common Sci-Fi trope of the super intelligent robot that struggles to communicate with humans. It turns out we've created something that sounds credible and smart and mostly human well before we made something with actual artificial intelligence.
I don't know exactly what to make of that inversion, but it's definitely interesting. Maybe it's just evidence that fooling people into thinking you're smart is much easier than actually being smart, which certainly would fit with a lot of events involving actual humans.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#187Why don't people here on HN understand that LLMs never see ASCII or other raw characters as input? Expecting spelling, rhyming, arithmetic or other character oriented responses will always yield disappointing results.
We do understand. We don't think that's okay. If a model cannot manage character level consideration, that's a serious flaw that's got potential to lead to an immeasurable number of failure states. "Duh, of course it can't count" is not the best look for a bot whose author tells us it's got PhD-level skill.
Can you enumerate some of these "immeasurable number of failure states"? For me it starts and stops at "can't count letters in a word". That hardly seems catastrophic.
All I have to do is turn on thinking mode and the error goes away. https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e271...
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#188A couple of weeks ago, I asked google, ordinary google search, how many times the letter r is found in preferred, and it told me 2. This century has taken quite a bitter turn against those of us who think that the 'enough' in 'good enough' ought to exclude products indistinguishable from the most grievously disgraceful products of sloth. But I have also lately realized that human beings, brains, society, culture, edu…
The generation leading this world has all the weapons the previous generation built at their disposal and none of the discipline or education to wield them responsibly. This too shall pass, but how it passes will be interesting to see.
You could say that about any two adjacent generations. Nuclear weapons are already 3 generations ago.
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#189This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.
Sure, but I think the point is why do LLM's have a blindspot for performing a task that a basic python script could get right 100% of the time using a tiny fraction of the computing power? I think this is more than just a gotcha. LLMs can produce undeniably impressive results, but the fact that they still struggle with weirdly basic things certainly seems to indicate something isn't quite right under the hood. I have…
Hence positronic neural network outperforms machine learning that are used today. /headduck
Re: GPT-5: "How many times does the letter b appear in blueberry?"
#190Good to see this can still be used as a "skill-testing" question for anti-bot defenses, which doesn't require bending over to Big Tech and Big Browser.