Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

41–50 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#41

Post by wqaatwt »

reliablereason wrote: post #31

Is the thinking even done in real tokens? I thought it was done using the pure residual stream. That is instead of collapsing the residual stream to a token you treat the final layers output as a vector of size d_model and use that as input for the next position in the transformer. If that is the case thinking is not visible to us as users due to it not being done in text.

All open model that have reasoning seem to be doing it in text tokens. Is there any indication that closed models are approaching this somehow fundamentally differently?

Re: The text in Claude Code’s “Extended Thinking” output

#42

Post by 0o_MrPatrick_o0 »

stingraycharles wrote: post #34
craigmart wrote: post #24

This is something we have known for a very long time, and companies are not trying to hide that either. They do it to avoid letting competitors train their models on the CoTs

Yes hasn’t this been around since Opus 4.6? I very much recall this change happening around January or February, and it was very explicitly to prevent distillation. Sonnet does not have this limitation. Fun fact: if you go back to the old school from 2 years ago and provide explicit CoT prompts, you get the full thinking prompts back again! So you disable thinking altogether, and instead make thinking part of the reg…

Awesome share! Thank you!

Re: The text in Claude Code’s “Extended Thinking” output

#43

Post by InsideOutSanta »

wqaatwt wrote: post #39

Is this some new revelation? That was well known when the first OpenAI/Anthropic “thinking” models came out.

It's not a new revelation, but clearly a lot of people aren't aware of it, so talking about it is still valuable.

Re: The text in Claude Code’s “Extended Thinking” output

#44

Post by Sharlin »

StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

The cynic in me is wondering whether it's more about how revealing how the sausage is made might bring bad publicity.

Re: The text in Claude Code’s “Extended Thinking” output

#45

Post by anuramat »

Closi wrote: post #28
anuramat wrote: post #16

Earlier quoted context omitted.

how is summarized CoT a moat, and how is having the top 2 LLMs not a moat?

If you have the full outputs, it might make it easier for competitors to distil the model or reverse engineer the full process. It may also be that misaligned responses can be in CoT which OpenAI does not want to show to users.

but "harder to reverse engineer" isn't manufacturing, that's protecting your moat

Re: The text in Claude Code’s “Extended Thinking” output

#46

Post by shideneyu »

StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

correct. this becomes difficult for us to understand what happens behind the scenes.

Re: The text in Claude Code’s “Extended Thinking” output

#47

Post by InsideOutSanta »

philipwhiuk wrote: post #26

To be honest I thought the 'thinking' was the model being asked 'how did you come up with that' and then it generating a plausible explanation. I know at one point this was correct. Humans somewhat do the same - something that's been demonstrated in split-brain experiments.

If you ask an LLM afterward how it arrived at an answer, it might produce a plausible but incorrect explanation. But that's not what the thinking stream is; that's actually part of how it generates the answer.

Re: The text in Claude Code’s “Extended Thinking” output

#48

Post by [deleted] »

Sharlin wrote: post #44
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

The cynic in me is wondering whether it's more about how revealing how the sausage is made might bring bad publicity.

[deleted]

Re: The text in Claude Code’s “Extended Thinking” output

#49

Post by bigfishrunning »

Sharlin wrote: post #44
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

The cynic in me is wondering whether it's more about how revealing how the sausage is made might bring bad publicity.

Imagine if their target customers, C-suite execs looking to replace workers, knew how unlike "thinking" this process actually was! we can't have that.

Re: The text in Claude Code’s “Extended Thinking” output

#50

Post by throwuxiytayq »

reliablereason wrote: post #31

Is the thinking even done in real tokens? I thought it was done using the pure residual stream. That is instead of collapsing the residual stream to a token you treat the final layers output as a vector of size d_model and use that as input for the next position in the transformer. If that is the case thinking is not visible to us as users due to it not being done in text.

That would be a huge deal, meaning we've lost even our shitty, ineffective ways of monitoring agent reasoning stream. Big setback when it comes to alignment and interpretability.

I don't know about Claude, but latest GPT versions still have a readable reasoning stream. It sometimes leaks out when the model gets confused, e.g., during a tool call. If you're curious, looks simplified; less words; extremely compact. They optimize tokens. But remain readable.

Post reply on HN