Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Reasoning Traces from Proprietary LLM APIs

stolen-thoughts.com

Post reply on HN
Print view View original

221–230 of 325 posts

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#221

Post by [deleted] »

glub wrote: post #213
chrisss395 wrote: post #212

Earlier quoted context omitted.

Possibly something to do with other providers using the it to train their own models?

The only "secret" there is a very basic instruction that the model receives, like "summarize current state and upcoming work" before compaction - same model that was just running your inference, with same cache, only server side, with no extra tools or capabilities. Then the fresh context gets the output from that as an encrypted blob + codex then injects up to 64k tokens of previous conversation, the latter part is…

[deleted]

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#222

Post by ashirviskas »

Pragmata wrote: post #119

Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool... >guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right? >gl fixing that https://x.com/_can1357/status/2087228354399265125?s=20

I've been doing that since before reasoning was a thing baked into the models, it always performs better this way. Except for some providers/models where you just can't easily turn it off, now I just avoid them. This way I save tokens and have full control of the reasoning.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#223

Post by brianxq3 »

paxys wrote: post #153
Aissen wrote: post #138

"Stealing" something you already paid for (tokens), but that you can't have access to(!). And trained on the sum of human knowledge. Training on other model outputs ought to be business as usual, stop using morally charged terms made up by future monopolists: https://thomasdullien.github.io/posts/2026-06-15-rl-economic...

The only person calling it stealing is the author of this article, so this is a pointless discussion. The majority of this thread is just arguing with themselves.

They are also encrypting it so they must see some reason to do this. I suspect they think it is proprietary or otherwise a way that people can “steal” their implementations.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#224

Post by ashdksnndck »

Aissen wrote: post #138

"Stealing" something you already paid for (tokens), but that you can't have access to(!). And trained on the sum of human knowledge. Training on other model outputs ought to be business as usual, stop using morally charged terms made up by future monopolists: https://thomasdullien.github.io/posts/2026-06-15-rl-economic...

Suppose you hired a consulting firm to write a report, and they delivered the report but not the internal conversations they had when developing it. You exploit a vulnerability in their phone system to get those conversations. You can argue over semantics of whether “theft” is what you did, maybe the right word is “espionage” or “spying”, but that either way we probably agree you are guilty of something? Paying for the final product didn’t entitle you to see how it was made, unless that was part of the agreement.

Anyway, you can distinguish this from the debate over copyright.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#225

Post by varenc »

super interesting. So pre-filling Kimi3 reasoning with Opus's reasoning results in thoughts that closely match Opus's. This seems like strong evidence Kimi3 was trained on decrypted Opus chain-of-thought. Meaning the Kimi team likely also broke CoT encryption. Though not exactly a big surprise.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#226

Post by varenc »

glub wrote: post #213
chrisss395 wrote: post #212

Earlier quoted context omitted.

Possibly something to do with other providers using the it to train their own models?

The only "secret" there is a very basic instruction that the model receives, like "summarize current state and upcoming work" before compaction - same model that was just running your inference, with same cache, only server side, with no extra tools or capabilities. Then the fresh context gets the output from that as an encrypted blob + codex then injects up to 64k tokens of previous conversation, the latter part is…

The compaction prompt doesn't seem like the valuable thing here. I suspect they're protecting the compaction result itself. If you're trying to distill a model, collecting lots of examples on how a large conversation gets compacted to a smaller summary is particularly useful data.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#227

Post by varenc »

driverdan wrote: post #214
HarHarVeryFunny wrote: post #86

Earlier quoted context omitted.

For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.

This isn't a safety issue, it's LLM companies trying to be opaque and stop distillation.

By some interpretations protecting your frontier model with strong safeguards from being distilled into an open source model without safeguards is a safety issue.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#228

Post by varenc »

desterothx wrote: post #60
NitpickLawyer wrote: post #25

Earlier quoted context omitted.

> But the real question is: Is it okay to steal By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :) I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for th…

I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on

This article shows that when Kimi3's chain of thought is prefilled to match Opus's, the rest of the chain of thoughts Kimi3 outputs very closely aligns with Opus's. That seems strong evidence that Kimi3 is partly a distillation of Opus. And Kimi3 is not a small model. No doubt a lot of hard work went into Kimi, but seems clear that distillation was used effectively as well.

(though maybe there's another interpretation of the thought alignment?)

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#229

Post by varenc »

tanh wrote: post #42

So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.

If CoT wasn't stateless and you instead just got a reference which pointed to the CoT stored on the lab servers, the same vulnerability would still exist. Since you just need a weaker jailbroken model to read a smarter model's CoT. This being stateless or not doesn't really matter.

The stateless part is also important for enterprise customers that require zero data retention.

(they could scope CoT access per model, but then users couldn't switch models mid-session)

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#230

Post by varenc »

throwa356262 wrote: post #61

This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return. Anyway, can someone explain the part about K3? What are they trying to say?

The part about K3 is just very strong evidence that K3 is partly a distillation of Opus. Probably even a distillation of Opus's CoT, which means they already had broken CoT encryption themselves awhile ago.
Post reply on HN