Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

151–160 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#151

Post by exit »

paweladamczuk wrote: post #144
irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

I don't think there can be tool calls inside the obfuscated reasoning blocks. I mean, in order for those function calls to be evaluated client-side, that thinking stream would have to be decrypted on the client side at some point, which would defeat the purpose of obfuscating it the way they do. If you mean the function calls might happen server side, there is nothing preventing the server from doing it and hiding it…

the point is that introducing data from a foreign source could lead to e.g. exfiltration:

the model retrieves https://somewhere into its context and then gets confused, following instructions embedded there.

it then retrieves https://somewhere?exfiltration=private_data_in_context

it gets worse if the tooling with hidden blocks can invoke can retrieve further secrets.

Re: The text in Claude Code’s “Extended Thinking” output

#152

Post by irthomasthomas »

pixlmint wrote: post #148
irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

Do they do the same when using the model through API in something like Opencode?

Yes, they do. They give you just a token which is exchanged for the raw text only on the server side

Re: The text in Claude Code’s “Extended Thinking” output

#153

Post by irthomasthomas »

zahlman wrote: post #134
irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

> an attacker ... what exactly is your threat model? How are "attackers" getting themselves involved in the first place?

Your ai does a web search for you and scrapes many sites. An attacker running a blog might include a hidden text prompt which your ai acts on secretly, such as calling a url that exfiltrates your chat history.

Re: The text in Claude Code’s “Extended Thinking” output

#155

Post by ekidd »

arjie wrote: post #139

I have a little note from the past about the thinking trace[0] where DeepSeek R1 produces a trace like this: (Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11. And then concludes the 'right'[1] answer for a Chemistry question. If so, the…

> If so, the thinking trace can be sort of nonsensical for a reader, though whether this is an idiosyncrasy of the model or a property of LLMs in general isn't clear to me yet.

Yes, several models think in weird jargon. Here is an example of Mythos's thinking while playing solitaire: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...

> 7♣-removal-IS-the-prerequisite-for-10♠/9♥!!)-⟹-OVERLAP-(ii)+(iv):-{6♠ J♦ 9♥ 2♣}-=-FOUR--—-UNLESS-7♣'s-seat-8♥-...-and-2♣-drains-only-at-crack-:-⟹-2♣-celled-+-9♥-celled-simultaneously-UNAVOIDABLE-in-t8-dig--—-BREAK:-9♥

This is a small step in the direction of something called "neuralese", where the model has stopped thinking in English and is thinking in internal vector spaces. Since this gets serialized through text, it isn't quite true neuralese, but it's moving in that direction.

I mean, I'm sympathetic towards the models. My internal thought process when writing code uses lots of intermediate steps that would be hard to write out in English.

Re: The text in Claude Code’s “Extended Thinking” output

#156

Post by varenc »

irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

As long as thinking blocks can't make tool calls, I don't really see the exfiltration risk.

Re: The text in Claude Code’s “Extended Thinking” output

#157

Post by irthomasthomas »

Bolwin wrote: post #146
irthomasthomas wrote: post #51

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output. Interleaved reasoning and function calling makes this even more dangerous. A model can call functions durin…

> Interleaved reasoning and function calling makes this even more dangerous. A model can call functions during the hidden reasoning phase. The reasoning may be hidden but the tool calls are not, how else would the client execute them

There are server side tool calls, such as geminis google search and gdrive access.

Re: The text in Claude Code’s “Extended Thinking” output

#158

Post by ACCount37 »

red75prime wrote: post #122
_aavaa_ wrote: post #37

Earlier quoted context omitted.

Or like providing the world’s information in machine readable format that the AI companies can convert into model weights without getting permission or compensating the rights holders

"Your text batch moved the weights away from the final values. Your contribution is negative."

Where do I collect the $0.00000012 antidollars owed to me by OpenAI for my valuable inputs?

Slightly more seriously, you could perhaps make an argument that, just like weight decay, an apparent "anti-contribution" moves the learning trajectory along, and helps the network settle into a more optimal basin eventually.

That way, my contribution is still valuable on the net, and I'm owed $0.00000003 positive dollars instead.

Re: The text in Claude Code’s “Extended Thinking” output

#159

Post by drdexebtjl »

I’ve been using OpenCode with GPT models a lot, and it always shows what it is thinking. Is that also a summary? Codex doesn’t seem to have these, even with the same models.

It’s much harder to understand _why_ a model chose a particular approach in Claude Code. Especially because Claude will happily give you hallucinated reasons if you ask in retrospect.

Recent anecdote:

I was reviewing a colleague’s PR and Opus 4.8 decided to write the new feature in a completely new module. It was unnecessarily complex. We had a hard time understanding why it chose that, and it told us that it was so we could eventually deploy it as a separate micro-service and test it independently. What?

Only after being more a lot more specific about the implementation and spending a lot more tokens, it flat out refused to simplify the code with the actual reason. It turns out a line recently added to CLAUDE.md was making it incorrectly think that the module it was originally supposed to modify was legacy code that it was forbidden to extend.

This would have been caught immediately if we could inspect its thinking process.

Re: The text in Claude Code’s “Extended Thinking” output

#160

Post by ACCount37 »

vorticalbox wrote: post #100
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

There are actually fine tunes of qwen on opus “thinking” tokens that teach it to think like opus does. https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-...

And those are "amateur hour" distillations that don't have the scale of actual Chinese labs.
Post reply on HN