31–40 of 248 posts
If that is the case thinking is not visible to us as users due to it not being done in text.
Not surprised at this. The questoins for enterprises are + where can you depend on a black box as a service? + what evals and observability do you need to deploy a black box as a service confidently? + what's the ROI (considering a total footprint of people, token spend, infrastructure, service, ops etc.) The LLM providers will clearly evolve to be more and more opaque as their services get more capable. The frontier…
This is something we have known for a very long time, and companies are not trying to hide that either. They do it to avoid letting competitors train their models on the CoTs
Fun fact: if you go back to the old school from 2 years ago and provide explicit CoT prompts, you get the full thinking prompts back again!
So you disable thinking altogether, and instead make thinking part of the regular prompt by prompting it:
“Before providing your answer, think step by step. For example:
The use is asking me to… I need to think about the blah blah. First, I should foo the bar, and then blah blah.
Answer: ”
And tada.wav we have CoT as it worked in the GPT3 era back again.
Is the thinking even done in real tokens? I thought it was done using the pure residual stream. That is instead of collapsing the residual stream to a token you treat the final layers output as a vector of size d_model and use that as input for the next position in the transformer. If that is the case thinking is not visible to us as users due to it not being done in text.
This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of t…
To be honest I thought the 'thinking' was the model being asked 'how did you come up with that' and then it generating a plausible explanation. I know at one point this was correct. Humans somewhat do the same - something that's been demonstrated in split-brain experiments.
Because of the nature of how LLMs work — text prediction engines - by putting the explicit reasoning steps first, it improves the likelihood of the final answer (which then is being predicted based on the entire reasoning chain as input) being correct.
When you have no moat, you have to try and find desperate ways to manufacture one.
wdym?
This could all be optics as well to try to give the appearance of a defensible moat. E.g. they can claim to investors that they are able to protect a significant chunk of their intellectual property this way. I'm not sure if anyone has a study about how significant the summarization is to distillation.