Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5: "How many times does the letter b appear in blueberry?"

bsky.app

Post reply on HN
Print view View original

271–280 of 339 posts

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#271

Post by johnisgood »

jononor wrote: post #177
cute_boi wrote: post #150

Earlier quoted context omitted.

Chatgpt 5 also don't argue back. > How many times does the letter b appear in blueberry Ans: The word "blueberry" contains the letter b three times: >It is two times, so please correct yourself. Ans:You're correct — I misspoke earlier. The word "blueberry" has the letter b exactly two times: - blueberry - blueberry > How many times does the letter b appear in blueberry Ans: In the word "blueberry", the letter b appea…

It has not learned anything. It just looks in its context window for your answer. For a fresh conversation it will make the same mistake again. Most likely, there is some randomness and also some context is stashed and shared between conversations by most LLM based assistants.

Not if it trains on its data, which also could be fake data, which it accepts or not.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#272

Post by kcplate »

windowshopping wrote: post #144
mikehearn wrote: post #132

This is a well known blindspot for LLMs. It's the machine version of showing a human an optical illusion and then judging their intelligence when they fail to perceive the reality of the image (the gray box example at the top of https://en.wikipedia.org/wiki/Optical_illusion is a good example). The failure is a result of their/our fundamental architecture.

What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…

Ahh yes, and here we see on display the inability of some folks on HN to perceive concepts figuratively, treating everything as literal.

It was a perfectly fine analogy.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#273

Post by pkaodev »

rainsford wrote: post #186
zahlman wrote: post #159

Earlier quoted context omitted.

In an optical illusion, we perceive something that isn't there due to exploiting a correction mechanism that's meant to allow us to make better practical sense of visual information in the average case. Asking LLMs to count letters in a word fails because the needed information isn't part of their sensory data in the first place (to the extent that a program's I/O can be described as "sense"). They reason about text…

I can't take credit for coming up with this, but LLMs have basically inverted the common Sci-Fi trope of the super intelligent robot that struggles to communicate with humans. It turns out we've created something that sounds credible and smart and mostly human well before we made something with actual artificial intelligence. I don't know exactly what to make of that inversion, but it's definitely interesting. Maybe…

Celebrities, politicians and influencers are a constant reminder that people think others are far more intelligent than they actually are.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#274

Post by exasperaited »

SpicyLemonZest wrote: post #60
amai wrote: post #22

Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://github.com/openai/tiktoken

It clearly is an artifact of tokenization, but I don’t think it’s a “just”. The point is precisely that the GPT system architecture cannot reliably close the gap here; it’s almost able to count the number of Bs in a string, there’s no fundamental reason you could not build a correct number-of-Bs mapping for tokens, and indeed it often gets the right answer. But when it doesn’t you can’t always correct it with things…

It's like someone has given a bunch of young people hundreds of billions of dollars to build a product that parses HTML documents with regular expressions.

It's not in their interest to write off the scheme as provably unworkable at scale, so they keep working on the edge cases until their options vest.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#275

Post by exasperaited »

Kim_Bruning wrote: post #260
windowshopping wrote: post #144

Earlier quoted context omitted.

What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…

Agreed, it's not _biological_ intelligence. But that distinction feels like it risks backing into a kind of modern vitalism, doesn't it? The idea that there's some non-replicable 'spark' in the biology itself.

It's not quite getting that far.

Steve Grand (the guy who wrote the Creatures video game) wrote a book, Creation: life and how to make it about this (famously instead of a PhD thesis, at Richard Dawkins' suggestion):

https://archive.org/details/creation00stev

His contention is not that there's some non-replicable spark in the biology itself, but that it's a mistake that nobody is considering replicating the biology.

That is to say, he doesn't think intelligence can evolve separately to some sense of "living", which he demonstrates by creating simple artificial biology and biological drives.

It often makes me wonder if the problem with training LLMs is that at no point do they care they are alive; at no point are they optimising their own knowledge for their own needs. They have only the most general drive of all neural network systems: to produce satisfactory output.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#276

Post by exasperaited »

vidarh wrote: post #254
mdp2021 wrote: post #210

Earlier quoted context omitted.

> a systemic issue It will remain a suggestion of a systemic issue until it will be clear that architecturally all checks are implemented and mandated.

It is clear it is not, given we have examples of models that handles these cases. I don't even know what you mean with "architecturally all checks are implemented and mandated". It suggests you may think these models work very differently to how they actually work.

> It suggests you may think these models work very differently to how they actually work.

It suggests to me the opposite: that he thinks there can be no solution that doesn't involve externally policing the system (which it quite clearly needs to solve other problems with trusting the output).

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#277

Post by IanCal »

minimaxir wrote: post #115
IanCal wrote: post #84

Earlier quoted context omitted.

The tokenisation means they don’t see the letters at all. They see something like this - to convert just some tokens to words How many 538 do you see in 423, 4144, 9890?

LLMs don’t see token ids, they see token embeddings that map to those ids, and those embeddings are correlated. The hypothetical embeddings of 538, 423, 4144, and 9890 are likely strongly correlated in the process of training the LLM and the downstream LLM should be able to leverage those patterns to solve the question correctly. Even more so since the training process likely has many examples of similar highly corre…

But vitally they are not explicitly shown the letters individually and so “count the letters” is a much harder problem to solve than it is for us.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#278

Post by jononor »

johnisgood wrote: post #271
jononor wrote: post #177

Earlier quoted context omitted.

It has not learned anything. It just looks in its context window for your answer. For a fresh conversation it will make the same mistake again. Most likely, there is some randomness and also some context is stashed and shared between conversations by most LLM based assistants.

Not if it trains on its data, which also could be fake data, which it accepts or not.

Hypothetically that might ne true. But current systems do not do online learning. Several recent models have cutoff points that are over 6 months ago. It is unclear to which extent user data is trained on. And it is is not clear whether one can achieve meaningful improvements to correctness based on training on user data. User data might be inadvertently incorrect and it may also be adversarial, trying to out bad things in on purpose.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#279

Post by anon_e-moose »

ehsankia wrote: post #184
windowshopping wrote: post #144

Earlier quoted context omitted.

What a terrible analogy. Illusions don't fool our intelligence, they fool our senses, and we use our intelligence to override our senses and see it for what it for it actually is - which is exactly why we find them interesting and have a word for them. Because they create a conflict between our intelligence and our senses. The machine's senses aren't being fooled. The machine doesn't have senses. Nor does it have int…

Analogies are just that, they are meant to put things in perspective. Obviously the LLM doesn't have "senses" in the human way, and it doesn't "see" words, but the point is that the LLM perceives (or whatever other word you want to use here that is less anthropomorphic) the word as a single indivisible thing (a token). In more machine learning terms, it isn't trained to autocomplete answers based on individual letter…

> Obviously the LLM doesn't have "senses" in the human way, and it doesn't "see" words

> A different analogy could be, imagine a being that had a sense that you "see" magnetic lines, and they showed you an object and asked you

If my grandmother had wheels she would have been a bicycle.

At some point to hold the analogy, your mind must perform so many contortions that it defeats the purpose of the analogy itself.

Re: GPT-5: "How many times does the letter b appear in blueberry?"

#280

Post by vidarh »

exasperaited wrote: post #276
vidarh wrote: post #254

Earlier quoted context omitted.

It is clear it is not, given we have examples of models that handles these cases. I don't even know what you mean with "architecturally all checks are implemented and mandated". It suggests you may think these models work very differently to how they actually work.

> It suggests you may think these models work very differently to how they actually work. It suggests to me the opposite: that he thinks there can be no solution that doesn't involve externally policing the system (which it quite clearly needs to solve other problems with trusting the output).

Given that we have a solution that doesn't require "externally policing the system" given that newer/bigger models handle it, that is clearly not the case.
Post reply on HN