Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Reasoning Traces from Proprietary LLM APIs

stolen-thoughts.com

Post reply on HN
Print view View original

21–30 of 325 posts

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#21

Post by Der_Einzige »

yojo wrote: post #9
Groxx wrote: post #4

> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seem…

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

100% guaranteed that this research just forced this to happen now.

Sucks.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#24

Post by Groxx »

the_af wrote: post #6
quantumgarbage wrote: post #2

Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model di…

Why do you restate the abstract? Anyone can read it from the link.

It's rather common for posters to make a very small summary in a comment. It can help fight the floods of comments working off the title alone (though it's not particularly needed here for that purpose, imo)

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#25

Post by NitpickLawyer »

myworkaccount2 wrote: post #17

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

> But the real question is: Is it okay to steal

By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)

I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.

(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#26

Post by SwellJoe »

"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.

And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#27

Post by myworkaccount2 »

yojo wrote: post #9
Groxx wrote: post #4

> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seem…

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves?

This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.

If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.

For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#29

Post by adrian_b »

Der_Einzige wrote: post #20

The problem with this kind of excellent work is that the response to it is always to say "Fuck the user". For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the choppi…

What I found the most interesting, and unfortunately not at all surprising, is that the reasoning of the LLMs frequently contained much more useful information than the actual answers, because the answers were censored.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#30

Post by yubblegum »

x312 wrote: post #14

Super cool that this works. I'm surprised these companies re-use the same encryption key across models! I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888

Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI.

(Thanks for the link. That’s an interesting idea!)

Post reply on HN