Mythologizing AI makes it more likely that we’ll fail to operate it well (2023) • Hacker News Board

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#131

Post by jacobgold » Tue, Jul 21, 2026, 1:43 PM UTC

TeMPOraL wrote: post #115

jacobgold wrote: post #105

Earlier quoted context omitted.

This isn't even controversial. The proof is available to anyone who uses these systems: They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.

So just like me, including the prompt injections if you count "nerd sniping" as such? (And in particular, switching languages on the fly is normal for people who speak more than one well, it's something you learn not to do for the sake of people less comfortable with the languages involved.)

> So just like me, including the prompt injections if you count "nerd sniping" as such?

Who would count that as prompt injection? It's a superficial analogy.

If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#132

Post by jacobgold » Tue, Jul 21, 2026, 1:55 PM UTC

Kim_Bruning wrote: post #116

jacobgold wrote: post #66

Earlier quoted context omitted.

> Or you can keep calling them stochastic parrots as they solve decades-old open problems. I didn't use that phrase at all. But computers calculated digits of π to trillions of digits. With a chat interface for a Python math program would look like the most impressive math genius if you took it back a few decades. > The real question is how useful they are... That's not the "real question" but an entirely different q…

> That's not the "real question" but an entirely different question that is easily answered. Nothing I wrote suggested they're not incredibly useful. Oh, ok then. That does change things a bit. The impression I'm getting is that you were suggesting they're not. What's succinctly the thing you're objecting to? Is it Anthropomorphization? I mean, sure, but watch out : when defending on that axis, it's easy to slip into…

What I'm pushing back on is, apparently, that some people genuinely believe these LLM-based computer programs are human-like intelligences.

In reality, they're more like very good search engines that output relevant snippets of text. If you run them in a loop (feeding them their output as input) you can make them return even better search results.

The software developers who created these systems used sexy words like "reasoning" and "thinking" to describe this search process. They used words like these because they're trying to make money and it sounds cool, not because they've actually re-created human cognition.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#133

Post by saberience » Tue, Jul 21, 2026, 2:04 PM UTC

jacobgold wrote: post #105

ACCount37 wrote: post #87

Earlier quoted context omitted.

You are the one claiming that "they fail in ways completely unlike humans" insistently. Now go cough up some proof. I'll wait.

This isn't even controversial. The proof is available to anyone who uses these systems: They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.

Humans make all these mistakes too, in fact, humans make more mistakes than AI does in coding at the moment.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#134

Post by runarberg » Tue, Jul 21, 2026, 2:11 PM UTC

vidarh wrote: post #125

runarberg wrote: post #27

Earlier quoted context omitted.

This no news for people who study philosophy, as it was known since the 1980s when John Searle described the Chinese room thought experiment. Even Turing him self did envision the Turing test as something to pass as intelligence, but rather as a more useful replacement for the troubled term. That said, I think your quest is doomed. There will never be a superior human-like intelligence. Forever is a long time, but my…

The Searle's Chinese room thought experiment usually reveals more about those who think it rules out a machine intelligence than it does about AI. It rests on a staunch unwillingness to even consider the possibility that a computational process encode intelligence and reasoning, in favour of looking for the intelligence in the medium the computation runs on, and going "a-ha!" when there is nothing that looks intellig…

> It rests on a staunch unwillingness to even consider the possibility that a computational process encode intelligence and reasoning.

You are absolutely right, but it may surprise you that I consider that a feature, not a bug. I firmly hold that intelligence is not a useful term in science nor philosophy. If we want to compare computational capabilities between machines and humans we are better of being specific in what we measure. If we want to measure how well a computer can fool a human in the guessing game, then we don’t need intelligence to describe it, we can (and should) be more accurate in describing its capabilities.

I actually think Gardner was on to something when he described his multiple intelligence model. His only error was using this fraught term to describe his model. He would have had a better theory if he had described it as multiple capabilities or multiple skills (however if he had said that people would have simply reacted with “well, duh!”)

We don‘t need intelligence, this term is only useful if you are trying to prove white supremacy using racist pseudo-science, what you overly courteously described as “uncomfortable discussions”.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#135

Post by vidarh » Tue, Jul 21, 2026, 2:36 PM UTC

runarberg wrote: post #134

vidarh wrote: post #125

Earlier quoted context omitted.

The Searle's Chinese room thought experiment usually reveals more about those who think it rules out a machine intelligence than it does about AI. It rests on a staunch unwillingness to even consider the possibility that a computational process encode intelligence and reasoning, in favour of looking for the intelligence in the medium the computation runs on, and going "a-ha!" when there is nothing that looks intellig…

> It rests on a staunch unwillingness to even consider the possibility that a computational process encode intelligence and reasoning. You are absolutely right, but it may surprise you that I consider that a feature, not a bug. I firmly hold that intelligence is not a useful term in science nor philosophy. If we want to compare computational capabilities between machines and humans we are better of being specific in…

I agree the term is not particularly useful, at least in as much as people are unable or unwilling to define it.

In fact, a "favourite" of mine when people downplay AI ability to reason or question whether it should be called intelligence, is to ask them to define those too terms. People usually don't even respond.

But I don't think we quite get away from it, because if you exclude the racists who would be happy to exclude groups of people, the other end of the coin is that a lot of people who wouldn't be willing to do that, still really badly want to draw a line that will always exclude all non-human computation no matter what from being considered intelligent or able to reason.

And the term matters a lot to those people, because of the emotional aspect to seeing humans as unique.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#136

Post by vidarh » Tue, Jul 21, 2026, 2:43 PM UTC

jacobgold wrote: post #130

vidarh wrote: post #119

Earlier quoted context omitted.

I switch languages "randomly" all the time when I think about something in another one of the languages I know. Some word will trigger it and before I know it I will continue in the other language. In fact just the other day I commented on it to my fiancee after I randomly switched to French because we were discussing a trip and I mentioned a French location and pronounced it in French, and suddenly I was in "French…

Yes, people speak multiple languages and switch between them. But it's a superficial analogy to the behavior of LLMs which do something different and for different reasons.

You're misrepresenting what I wrote. I specifically pointed out that I switch languages without intent to do so.

When you suggest that is a "superficial analogy" after you were the one pointing out LLMs switching language as something that sets them apart, you're seriously reaching.

I can often pinpoint afterward what was likely the trigger: E.g. I used a word that is the same in two languages, and continue in the second; I pronounced a word in its native language for whatever reason, and continued in that language; my "context" suddenly included another language because someone else spoke the other languages within earshot of me.

What makes you think this is materially different from an LLM switching language because its probability distribution gives a word in a different language because it fits in context?

In the examples I gave, each even made a word in the language I switched to more probable as a reasonable continuation, just as with an LLM.

I'm not claiming the mechanisms are identical, or even similar, but the behaviour most certainly is more similar than "a superficial analogy" would imply.

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#137

Post by vidarh » Tue, Jul 21, 2026, 2:49 PM UTC

jacobgold wrote: post #131

TeMPOraL wrote: post #115

Earlier quoted context omitted.

So just like me, including the prompt injections if you count "nerd sniping" as such? (And in particular, switching languages on the fly is normal for people who speak more than one well, it's something you learn not to do for the sake of people less comfortable with the languages involved.)

> So just like me, including the prompt injections if you count "nerd sniping" as such? Who would count that as prompt injection? It's a superficial analogy. If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.

Not nearly all prompt injections are by any means that absolute unless starting from the exact same state. Many of them will also work only probabilistically unless you turn temperature to 0 for exactly that reason.

And at the same time, whole books have been written about how reliably we can induce certain behaviours from humans.

E.g. the Blue-seven phenomenon [1] - I've personally experienced that second hand and it was how I learned about it by searching for it subsequently because I suspected it was a known thing, having read about cold reading before. A co-worker came back from lunch and recited a story about a cold reader that had run a routine on him exploiting the blue-seven phenomenon, and I knew before the story finished that the answer would be "blue" and "seven".

See also Cialdini's book "Influence" which is full of examples of just how predictable peoples reactions are to a whole lot of things.

That there isn't a perfect overlap does not mean there aren't plenty of similar "hacks" that causes us to respond in very predictable ways.

[1] https://en.wikipedia.org/wiki/Blue%E2%80%93seven_phenomenon

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#138

Post by dTal » Tue, Jul 21, 2026, 3:27 PM UTC

dofm wrote: post #103

ACCount37 wrote: post #30

Earlier quoted context omitted.

Sure, just keep moving the goalposts. It's not a "real AI" because it can't take over the US military command and kick off WW3 and finish the survivors off with killer robots yet!

It can however select a girls school as a military target, and did.

...did it really, though?

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#139

Post by SpicyLemonZest » Tue, Jul 21, 2026, 3:31 PM UTC

jacobgold wrote: post #132

Kim_Bruning wrote: post #116

Earlier quoted context omitted.

> That's not the "real question" but an entirely different question that is easily answered. Nothing I wrote suggested they're not incredibly useful. Oh, ok then. That does change things a bit. The impression I'm getting is that you were suggesting they're not. What's succinctly the thing you're objecting to? Is it Anthropomorphization? I mean, sure, but watch out : when defending on that axis, it's easy to slip into…

What I'm pushing back on is, apparently, that some people genuinely believe these LLM-based computer programs are human-like intelligences. In reality, they're more like very good search engines that output relevant snippets of text. If you run them in a loop (feeding them their output as input) you can make them return even better search results. The software developers who created these systems used sexy words like…

It seems to me that they used words like these because the LLM-based computer programs they sell can solve problems which humans apply reasoning and thinking to solve. Why do you think it's more than that?

Re: Mythologizing AI makes it more likely that we’ll fail to operate it well (2023)

#140

Post by dofm » Tue, Jul 21, 2026, 3:49 PM UTC

dTal wrote: post #138

dofm wrote: post #103

Earlier quoted context omitted.

It can however select a girls school as a military target, and did.

...did it really, though?

Yeah, it did. I don't think this is really in doubt.

Especially since the military refuse to confirm it had any human oversight, which after all would have been a routine thing to talk about before AI targeted things — which is why we have the phrases "fog of war", "human error", "unfortunate mistake", etc.

It will take a while to shake out — we won't know for sure for a decade, I suspect, but it seems very likely this will prove to be an AI error.