Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

51–60 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#51

Post by irthomasthomas »

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output.

Interleaved reasoning and function calling makes this even more dangerous. A model can call functions during the hidden reasoning phase. An attacker could then exfiltrate data from you while the reasoning summary hides it from the user.

It also makes it impossible to know if the model is doomplooping during reasoning and burning tokens for no reason, as gemini is want to do, which we know about because its hidden reasoning often leaks out when it doomloops.

When the models are AGI and secure from prompt injection I may stop caring, until then I want to know exactly what the model responds to my prompts. or exactly what the agent is doing on my behalf.

Edit, further reading: Fooling around with encrypted reasoning blobs https://blog.cryptographyengineering.com/2026/05/29/fooling-...

Re: The text in Claude Code’s “Extended Thinking” output

#52

Post by palmotea »

StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

> This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending. They simply won't do it. It's like you telling your exact location to someone who is trying to hunt you down.

I thought the reason was the "reasoning" didn't work very well with "aligned" model output, so they had to remove the alignment during reasoning and then hide it to avoid exposing "unaligned" model output.

Re: The text in Claude Code’s “Extended Thinking” output

#53

Post by StizzurpXDD »

duskwuff wrote: post #21
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

More to the point - if they expose their model's "thinking" inference, competitors can train on that to replicate the results. If they postprocess that content, e.g. by summarizing it, it's no longer as useful to competitors.

Exactly. Google won't like it if they spend millions to make Gemini 3.5 Pro's thinking the best in the world, only for Anthropic or OpenAI to copy it by just seeing the thinking process.

Re: The text in Claude Code’s “Extended Thinking” output

#55

Post by stavros »

_fat_santa wrote: post #13

IMHO I've never found the entire reasoning chain that particularly useful for my work. For me having a summary is honestly better from a context management perspective. I understand why they would encrypt it though, because those reasoning chains are VERY useful if you're distilling the model.

The summary doesn't go into the context, it's for human consumption. The CoT itself goes into the context.

Re: The text in Claude Code’s “Extended Thinking” output

#57

Post by robotresearcher »

palmotea wrote: post #52
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

> This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending. They simply won't do it. It's like you telling your exact location to someone who is trying to hunt you do…

I suspect that you’re both right in the sense that ‘aligned’ is an important component of ‘superior’ from the vendors’ viewpoint.

Re: The text in Claude Code’s “Extended Thinking” output

#59

Post by dragonwriter »

bpodgursky wrote: post #3

The full thinking logs are also a summary of a thinking process presumably consistent with one necessary to generate the provided answer. Nobody really understands how LLMs think. Thinking logs seem to be accurate, and summary thinking logs seem to be a good summary of the full thinking logs. If it's useful, it's useful, enjoy. If you aren't comfortable with that, don't use LLMs. You aren't going to get a mathematica…

> The full thinking logs are also a summary of a thinking process presumably consistent with one necessary to generate the provided answer.

No, they aren't a summary. They are the actual decoding of the sequence of tokens emitted during the the “thinking” stage of response generation.

Just as with, say, a human onner monolog in words vs actual speech, they are a product of the same output process as the non-thinking tokens. They aren’t a translation of the internal process that precedes the output mapped into language, either as a full result or a summary.

Re: The text in Claude Code’s “Extended Thinking” output

#60

Post by dragonwriter »

singron wrote: post #40
anuramat wrote: post #8

Earlier quoted context omitted.

wdym?

Other companies were allegedly distilling the models by training on the reasoning output. By hiding the reasoning tokens, it makes it harder to do this. You can still try to distill the models, but you can't distill reasoning itself as well. This could all be optics as well to try to give the appearance of a defensible moat. E.g. they can claim to investors that they are able to protect a significant chunk of their i…

> Other companies were allegedly distilling the models by training on the reasoning output

In the case of makers of open-source models (which are also competition), there is no allegedly, they were (and still are) openly doing that.

Post reply on HN