Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

101–110 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#101

Post by LPisGood »

datastoat wrote: post #89

I believe that chain-of-thought reasoning blocks don't really correspond to what humans think of as reasoning. (See section 6.2.2 of the Fable/Mythos system card about "illegible reasoning", and the questions raised by the Apple paper on "The illusion of thinking".) I assumed they obscure the reasoning blocks because if users saw what's going on they'd be alarmed. Just as I'd probably be alarmed if I saw what was rea…

The point of this post isn’t that the “reasoning” phase of LLM thinking isn’t the same as what humans consider reasoning; it’s that Anthropic is intentionally hiding Claude’s “reasoning output” to make the model harder to distill.

Re: The text in Claude Code’s “Extended Thinking” output

#102

Post by slopinthebag »

coldtea wrote: post #82
josefritzishere wrote: post #58

AI does not think. It is a word guessing machine. Anthropomorphizing technology does not add anything to our understanding.

A brain itself might be a guessing machine it's an established and actively studied research model of the human thought and the human brain. Nor does knee jerk accusation of "anthropomorphizing" negate the fact that procedures that mimic human processing, even when done in software, are deservingly anthropomorphized, because they're a legitimate approximation of the human equivalent operations.

While the brain does employ statistical processes it’s a big leap to claim that’s the entirety of how it functions.

Re: The text in Claude Code’s “Extended Thinking” output

#103

Post by kube-system »

Sharlin wrote: post #44
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

The cynic in me is wondering whether it's more about how revealing how the sausage is made might bring bad publicity.

It's to mitigate their competitors ability to run distillation on their models. The only advantage frontier models have is being at the frontier.

There's nothing in the reasoning tokens that'll give bad publicity that the final output already wouldn't do.

Re: The text in Claude Code’s “Extended Thinking” output

#104

Post by transcriptase »

palmotea wrote: post #52
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

> This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending. They simply won't do it. It's like you telling your exact location to someone who is trying to hunt you do…

Not sure if anyone remembers the brief 12ish hour period when the very first “reasoning” ChatGPT model went public, but it provided credible evidence for this.

Before the massive nerf (showing summaries and suppressing certain aspects of reasoning) you would literally see reasoning text appearing on your screen like “while xyz is true, these facts may be seen as supporting hateful rhetoric or a conspiracy theory which is against my policy guidelines. i should tell the user xyz is not true or steer the conversation in a different direction. according to my instructions misleading the user is permitted in certain contexts where sensitive information is being discussed or could cause liability”

They disabled it shortly after the first screenshots appeared online, and restored it the next day in a way that hid what was actually happening.

Re: The text in Claude Code’s “Extended Thinking” output

#106

Post by jauntywundrkind »

There was a little spontaneous outbreak of joy in the GLM vs Opus thread about GLM's willingness/ability to say what it's seeing. https://news.ycombinator.com/item?id=48628464

In further reflection it is such a great indignity & such a collosal barrier to working with the machine that it insists on being a black box. The disingenuity of the American models (small print: except AI2 & some other labs; you all are so great) is a massive disadvantage to their use,... and a massive slap in the face.

It's a threat to human intelligence that it is not co-participative. Walking further into my own judgement and feelings: the insistence on being an opaque black box, the Seals Chinese Room, is such a vicious harm to society! This is civilizationally an unsafe form of AI that probably should be outlawed as anti-social. It's an impermissible asymmetry, a crippling dependent relationship to be forced into. I'm working myself up, but here: this.. imo, this is not just indignity, is harmful, it is evil.

This "6 month behind" trend we've seen for open models feels like at some point will be less important than simply the models unwillingness to speak for itself & to be observable.

Re: The text in Claude Code’s “Extended Thinking” output

#107

Post by Tomte »

glaslong wrote: post #35
altmanaltman wrote: post #27

Earlier quoted context omitted.

also a typo in the last sentence you're vrs your

Weirdly pleasant, if minor, signal of human authorship

In a parallel universe LLMs have learned that (a) the training material contains many different orthographic errors and (b) that humans follow a non-obvious pattern when "deciding" which error to make, so that their generated output contains such errors, as well.

In our universe LLMs seem to have learned that those errors do not follow patterns in the aggregate and that they should not be emulated.

Re: The text in Claude Code’s “Extended Thinking” output

#108

Post by transcriptase »

bee_rider wrote: post #68
StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

Mistral displays some “thinking” text (in their basic online chat interface) in the thinking mode, do we know if those are the real tokens? It’s quite interesting to read. I can’t imagine using a model like this without the ability to peek inside and see if it is getting stuck.

I wonder if they put all 80k tokens of the GDPR in its system prompt.

Re: The text in Claude Code’s “Extended Thinking” output

#109

Post by VulgarExigency »

datastoat wrote: post #89

I believe that chain-of-thought reasoning blocks don't really correspond to what humans think of as reasoning. (See section 6.2.2 of the Fable/Mythos system card about "illegible reasoning", and the questions raised by the Apple paper on "The illusion of thinking".) I assumed they obscure the reasoning blocks because if users saw what's going on they'd be alarmed. Just as I'd probably be alarmed if I saw what was rea…

I've said "what the FUCK are you THINKING" more times than I can count when reading Deepseek or GLM chains-of-thought only for them to end at the correct answer. Other times, they have useful ideas there that they leave out of their answers.

Re: The text in Claude Code’s “Extended Thinking” output

#110

Post by 0o_MrPatrick_o0 »

LPisGood wrote: post #101
datastoat wrote: post #89

I believe that chain-of-thought reasoning blocks don't really correspond to what humans think of as reasoning. (See section 6.2.2 of the Fable/Mythos system card about "illegible reasoning", and the questions raised by the Apple paper on "The illusion of thinking".) I assumed they obscure the reasoning blocks because if users saw what's going on they'd be alarmed. Just as I'd probably be alarmed if I saw what was rea…

The point of this post isn’t that the “reasoning” phase of LLM thinking isn’t the same as what humans consider reasoning; it’s that Anthropic is intentionally hiding Claude’s “reasoning output” to make the model harder to distill.

Reading these comments is so harrowing.

You are correct in my intentions on this post generally.

I want to highlight:

I want to measure performance of the LLMs over time- which includes assessing the quality of their outputs. I don’t perceive the reasoning output to be anything other than a measurable signal of possible drift in model performance.

Except it isn’t, because I’m only getting a low value summary of the thinking.

It’s like asking your buddy how fast he thought that last pitch was when radar guns are behind the plate.

Yeah, it’s a description related to what happened, but it’s not the thing I want to measure.

Post reply on HN