Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The text in Claude Code’s “Extended Thinking” output

patrickmccanna.net

Post reply on HN
Print view View original

21–30 of 248 posts

Re: The text in Claude Code’s “Extended Thinking” output

#21

Post by duskwuff »

StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

More to the point - if they expose their model's "thinking" inference, competitors can train on that to replicate the results. If they postprocess that content, e.g. by summarizing it, it's no longer as useful to competitors.

Re: The text in Claude Code’s “Extended Thinking” output

#22

Post by HarHarVeryFunny »

This is nothing new - these companies don't want their model's output to be useful for distillation/training, so they just give a "summary" of its thinking steps rather than the actual sequence.

RL (the basis of LLM "thinking") is a pretty crude way to achieve the appearance of reasoning given that it reinforces all the steps, including missteps, that got it to a reward. Providing a summary could be seen as form of sane-washing, making the model look more purposeful and directed than it really is!

Re: The text in Claude Code’s “Extended Thinking” output

#25

Post by metadat »

StizzurpXDD wrote: post #18

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…

[dead]

Re: The text in Claude Code’s “Extended Thinking” output

#26

Post by philipwhiuk »

To be honest I thought the 'thinking' was the model being asked 'how did you come up with that' and then it generating a plausible explanation. I know at one point this was correct.

Humans somewhat do the same - something that's been demonstrated in split-brain experiments.

Re: The text in Claude Code’s “Extended Thinking” output

#27

Post by altmanaltman »

0o_MrPatrick_o0 wrote: post #15
furyofantares wrote: post #12

> It isn’t the actual thinking that drove the model’s actions in a session- but a summary of the thinking logic. This is like using saving a jpeg as a .bmp and then editing the .bmp and presenting it as a .jpeg. The conversion produces data loss. You've got that backwards, .bmp is a lossless format and .jpeg is the lossy one.

My bad! 10 points for House Slytherin!

also a typo in the last sentence you're vrs your

Re: The text in Claude Code’s “Extended Thinking” output

#28

Post by Closi »

anuramat wrote: post #16
ur-whale wrote: post #9

Earlier quoted context omitted.

> wdym? https://en.wikipedia.org/wiki/Economic_moat

how is summarized CoT a moat, and how is having the top 2 LLMs not a moat?

If you have the full outputs, it might make it easier for competitors to distil the model or reverse engineer the full process.

It may also be that misaligned responses can be in CoT which OpenAI does not want to show to users.

Re: The text in Claude Code’s “Extended Thinking” output

#29

Post by devmor »

philipwhiuk wrote: post #26

To be honest I thought the 'thinking' was the model being asked 'how did you come up with that' and then it generating a plausible explanation. I know at one point this was correct. Humans somewhat do the same - something that's been demonstrated in split-brain experiments.

That's not really how LLMs work at all. I would really recommend checking out something like [1] to get a rough understanding and avoid attributing too much to them.

1. https://medium.com/@eshvargb/the-llm-journey-how-neural-netw...

Post reply on HN