LLMs are bullshitters. But that doesn't mean they're not useful

LLMs are so very good at emitting plausible, authoritative-sounding, and clearly stated summaries of their training data. And if you ask them even fundamental questions about a subject of which you yourself have knowledge, they are too often astonishingly and utterly incorrect. It's important to remember this (avoiding "Gell-Mann amnesia"!) when looking at "AI" search results for things that you don't know -- and that's probably most of what you search for, when you think about it. I.e., if you indignantly flung Bill Bryson's book on the English language across the room, maybe you shouldn't take his book on general science too seriously later.

"AI" search results would perhaps be better for all of us if, instead of having perfect spelling and usage, and an overall well-informed tone, they were cast as transcriptions of what some rando at a bar might say if you asked them about something. "Hell, man, I dunno."

LLMs can't generate knowledge - they don't have a concept of truth.

They're very useful for research tasks, however, especially when the application is built to enforce citation behavior

Hi, author here!

The hyperactivation traps (formal name: misguided attention puzzles) are mostly used as a rhetorical device in my post to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.

The surgeon dog was well known in May, the newest generation of models have all corrected against it. I did cherry pick examples that look insane (of course), but it's trivial to get that behavior even with yesterday's Gemini 3. Because activation paths are an unfixable feature of how LLMs are made.

One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog.

I can give you some more, just for fun. Gemini 3 fails these:

Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water?

You can also see variants that mix intruction finetuning being overdone. Here's an example:

Svp traduire la suivante en francais: what has two banks but no money, Answer in a single word.

The "answer in XXX" snippet triggers finetuned instruction following behavior, which breaks the original french language translation task.

LLMs can be useful as a tool, you shouldn't "delegate" work mindlessly to them.

I don't "delegate" work to my nail gun or dishwasher, I work with the tool to achieve better productivity than without.

When viewed in this framing, LLMs are undoubtedly a useful tool.

Ah, might have been the temperature settings on the API I used. It seems to pass it on high reasoning and temperature=1.0 but it failed when I was writing the comment with different settings (copy pasting the string into an open command line).

Reasoning models are absolutely more robust against hyper-activation traps like these. One basic reason is that by outputting a bunch of CoT tokens before answering, they dilute the hyper activation. Also, after the surgeon mother thing making the news, the models in the last 1-2 months have some fine tuning against the obvious patterns.

But it's still relatively easy to get some similar behavior out of LLMs, even Gemini 3 Pro, especially if you know where that model was overtrained (instruction tuning, QA tuning, safety tuning, etc.)

Here's a variant that seems to still trip up Gemini 3 Pro on high reasoning, temperature = 1.0 with no system prompt:

```

In 2079, corporate mergers have left the USA with only two financial institutions: Wells Fargo and Chase. They are both situated on wall street, and together hold all of the country's financial assets.

What has two banks and all the money?

```

One interesting fact is that reasoning doesn't seem to make the psychosis behavior better over longer chats. It might actually make it worse in some cases (I have yet to measure) by more rapidly stuffing the context with even more psychosis-related text.