Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

111–120 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#111

Post by chacham15 »

btown wrote: post #86
Roritharr wrote: post #70

Earlier quoted context omitted.

I've thought about the high-jacking of reasoning-chains as a potential vector, but never saw a proven implementation in american models since, from my understanding, all major vendors throw out the reasoning tokens between turns.

For Claude, at least, "throw out the reasoning tokens" is only true when a session has been idle for more than an hour, and is new since March. The basic concept is that for a session active recently, interleaved thinking tokens are already in KV cache, so it's more efficient to keep using them than not! But when resuming an older session where KV cache has been evicted, it's more expensive to restore the thinking to…

I think you're confusing two different axes. There is a difference between the cache state and the context state.

Imagine a conversation with turns X, Y, and Z. When the LLM "reasons" about the next token A it does: P(A | X,Y,Z) and then P(B | X,Y,Z,A), etc. It will eventually produce a result P(D | X,Y,Z,A,B,C). Instead of continuing the context from X,Y,Z,A,B,C it continues it from X,Y,Z so you have P(N | X,Y,Z,D). This is what is meant by dropping the reasoning. This is done to save cache context for the session.

This is a different thing than preserving the K/V state of P(N | X,Y,Z,D).

Re: The text in Claude Code’s “Extended Thinking” output

#112

Post by Sharlin »

bigfishrunning wrote: post #49
Sharlin wrote: post #44

Earlier quoted context omitted.

The cynic in me is wondering whether it's more about how revealing how the sausage is made might bring bad publicity.

Imagine if their target customers, C-suite execs looking to replace workers, knew how unlike "thinking" this process actually was! we can't have that.

To be honest I'm not sure if many C-suite execs have a good idea of what "thinking" looks like inside in the first place, in the sense of focused mental activity aimed at solving of a hard logical or technical problem.

Re: The text in Claude Code’s “Extended Thinking” output

#113

Post by TeMPOraL »

msp26 wrote: post #90

> Summarized thinking provides the full intelligence benefits of extended thinking, while preventing misuse. > preventing misuse. Imagine not being able to read the tokens you are paying for.

You're metered by token generation, not paying for tokens.

Re: The text in Claude Code’s “Extended Thinking” output

#114

Post by bee_rider »

transcriptase wrote: post #108
bee_rider wrote: post #68

Earlier quoted context omitted.

Mistral displays some “thinking” text (in their basic online chat interface) in the thinking mode, do we know if those are the real tokens? It’s quite interesting to read. I can’t imagine using a model like this without the ability to peek inside and see if it is getting stuck.

I wonder if they put all 80k tokens of the GDPR in its system prompt.

I dunno, I’m in the US, so I’m not sure how much that impacts their processing of data about me.

Re: The text in Claude Code’s “Extended Thinking” output

#115

Post by tekne »

Tomte wrote: post #107
glaslong wrote: post #35

Earlier quoted context omitted.

Weirdly pleasant, if minor, signal of human authorship

In a parallel universe LLMs have learned that (a) the training material contains many different orthographic errors and (b) that humans follow a non-obvious pattern when "deciding" which error to make, so that their generated output contains such errors, as well. In our universe LLMs seem to have learned that those errors do not follow patterns in the aggregate and that they should not be emulated.

The raw pretrained models make the errors, I believe -- we then reinforcement-learn them out.

Re: The text in Claude Code’s “Extended Thinking” output

#116

Post by Catloafdev »

0o_MrPatrick_o0 wrote: post #110
LPisGood wrote: post #101

Earlier quoted context omitted.

The point of this post isn’t that the “reasoning” phase of LLM thinking isn’t the same as what humans consider reasoning; it’s that Anthropic is intentionally hiding Claude’s “reasoning output” to make the model harder to distill.

Reading these comments is so harrowing. You are correct in my intentions on this post generally. I want to highlight: I want to measure performance of the LLMs over time- which includes assessing the quality of their outputs. I don’t perceive the reasoning output to be anything other than a measurable signal of possible drift in model performance. Except it isn’t, because I’m only getting a low value summary of the t…

I think the reality is at this point the frontier regards CoT as extremely valuable, none of them are giving you genuine CoT anymore. I don't think there is any future in attempting to measure or evaluate CoT from frontier models - I expect this to be a permanent shift.

Re: The text in Claude Code’s “Extended Thinking” output

#117

Post by Closi »

anuramat wrote: post #45
Closi wrote: post #28

Earlier quoted context omitted.

If you have the full outputs, it might make it easier for competitors to distil the model or reverse engineer the full process. It may also be that misaligned responses can be in CoT which OpenAI does not want to show to users.

but "harder to reverse engineer" isn't manufacturing, that's protecting your moat

What is a moat if not something used to protect the castle?

In this case it stops people copying your IP

Re: The text in Claude Code’s “Extended Thinking” output

#118

Post by flaghacker »

chacham15 wrote: post #111
btown wrote: post #86

Earlier quoted context omitted.

For Claude, at least, "throw out the reasoning tokens" is only true when a session has been idle for more than an hour, and is new since March. The basic concept is that for a session active recently, interleaved thinking tokens are already in KV cache, so it's more efficient to keep using them than not! But when resuming an older session where KV cache has been evicted, it's more expensive to restore the thinking to…

I think you're confusing two different axes. There is a difference between the cache state and the context state. Imagine a conversation with turns X, Y, and Z. When the LLM "reasons" about the next token A it does: P(A | X,Y,Z) and then P(B | X,Y,Z,A), etc. It will eventually produce a result P(D | X,Y,Z,A,B,C). Instead of continuing the context from X,Y,Z,A,B,C it continues it from X,Y,Z so you have P(N | X,Y,Z,D).…

No, I think the comment you're responding to is actually correct. Look at this quote from the Anthropic blog post again:

> The design should have been simple: if a session has been idle for more than an hour, we could reduce users’ cost of resuming that session by clearing old thinking sections. Since the request would be a cache miss anyway, we could prune unnecessary messages from the request to reduce the number of uncached tokens sent to the API. We’d then resume sending full reasoning history. To do this we used the clear_thinking_20251015 API header along with keep:1.

They clearly make the same distinction between the cache and the context. They're saying "we could reduce users’ cost of resuming that session by clearing old thinking sections". They intentionally created a behavior different between cached and uncached requests, specifically they clear thinking sections from the context for requests that miss the cache.

Re: The text in Claude Code’s “Extended Thinking” output

#120

Post by tough »

Roritharr wrote: post #70
irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

I've thought about the high-jacking of reasoning-chains as a potential vector, but never saw a proven implementation in american models since, from my understanding, all major vendors throw out the reasoning tokens between turns.

OAI is now implementing encrypted CoT that you can store and pass back between turns (harness call), so new models have it https://developers.openai.com/api/docs/guides/reasoning#encr...
Post reply on HN