post title checks out
The text in Claude Code’s “Extended Thinking” output
131–140 of 248 posts
Re: The text in Claude Code’s “Extended Thinking” output
#132Is the thinking even done in real tokens? I thought it was done using the pure residual stream. That is instead of collapsing the residual stream to a token you treat the final layers output as a vector of size d_model and use that as input for the next position in the transformer. If that is the case thinking is not visible to us as users due to it not being done in text.
Thinking is implemented as regular autoregressive generations by everyone, meaning its just regular tokens, but they appear between special tokens which are then programmatically removed from what the user can actually see.
Idea somewhat similar to what you describe exist but they make steering/post-training/interpretation much harder.
Re: The text in Claude Code’s “Extended Thinking” output
#133Earlier quoted context omitted.
Other companies were allegedly distilling the models by training on the reasoning output. By hiding the reasoning tokens, it makes it harder to do this. You can still try to distill the models, but you can't distill reasoning itself as well. This could all be optics as well to try to give the appearance of a defensible moat. E.g. they can claim to investors that they are able to protect a significant chunk of their i…
> Other companies were allegedly distilling the models by training on the reasoning output In the case of makers of open-source models (which are also competition), there is no allegedly, they were (and still are) openly doing that.
In the case of the closed models too... Claude would happily tell you it was deepseek-v3 if you asked in chinese until it caught public attention and they papered over it.
Re: The text in Claude Code’s “Extended Thinking” output
#134I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…
> an attacker
... what exactly is your threat model? How are "attackers" getting themselves involved in the first place?
Re: The text in Claude Code’s “Extended Thinking” output
#135What I find sad is how much Anthropic goes to hide your data, yet they are happy to slurp up all yours and most of you are happy to hand it over. ... then they turn around and compete with you by building your products that eat into your market. Anthropic believes their reasoning tokens is a moat and that it's giving other labs an edge and that's why they are hiding it. If they really believe that is their edge, then…
I don't think people are happy to give it over, gullible and naive maybe?
Re: The text in Claude Code’s “Extended Thinking” output
#136Earlier quoted context omitted.
> all major vendors throw out the reasoning tokens between turns That would be surprising to me. The reasoning _is_ the model intelligence in a lot of respects, and so dropping those from the context would affect its output pretty significantly. I assume that instead they just have a lot of guardrails in place and multiple runtime environments that an individual turns ping-pong between in order to dehydrate/rehydrate…
Anthropic very explicitly says below their diagrams ( https://platform.claude.com/docs/en/build-with-claude/contex... ) on this: "Stripping extended thinking: Extended thinking blocks (shown in dark gray) are generated during each turn's output phase, but are not carried forward as input tokens for subsequent turns. You do not need to strip the thinking blocks yourself. The Claude API automatically does this for you…
https://platform.claude.com/docs/en/build-with-claude/extend...
default depends on the model class. Opus: Claude Opus 4.5 and later Opus models keep all prior thinking blocks; Claude Opus 4.1 (deprecated) and earlier Opus models keep only the last assistant turn's thinking. Sonnet: Claude Sonnet 4.6 and later Sonnet models keep all; Claude Sonnet 4.5 and earlier Sonnet models keep only the last turn. Haiku: all Haiku models through Claude Haiku 4.5 keep only the last turn. Claude Mythos Preview also keeps all prior thinking blocks.
Re: The text in Claude Code’s “Extended Thinking” output
#137[deleted]
Re: The text in Claude Code’s “Extended Thinking” output
#138Cat and mouse measures like this rarely work forever.
Re: The text in Claude Code’s “Extended Thinking” output
#139I have a little note from the past about the thinking trace[0] where DeepSeek R1 produces a trace like this:
(Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
0: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic#I...?
1: In the sense of true belief, I suppose
Re: The text in Claude Code’s “Extended Thinking” output
#140This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…
> Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending.
I think one of the reasons could be to limit liability too.
What if reasoning helps in establishing provenance for questionable sources ?
What if reasoning and model's "thought" points to fundamental issues in how the model was trained to produce certain problematic responses ?