Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Reasoning Traces from Proprietary LLM APIs

stolen-thoughts.com

Post reply on HN
Print view View original

31–40 of 325 posts

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#31

Post by aix1 »

yojo wrote: post #9
Groxx wrote: post #4

> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seem…

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#32

Post by tristanj »

azinman2 wrote: post #19
myworkaccount2 wrote: post #17

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.

No. There are dozens of companies that resell tokens at a discount to collect and resell session data to various Chinese labs.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#34

Post by tuesdaynight »

NitpickLawyer wrote: post #25
myworkaccount2 wrote: post #17

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

> But the real question is: Is it okay to steal By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :) I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for th…

It's like sideloading. It's very hard to fight against the marketing budget of big tech

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#35

Post by sandeepkd »

Groxx wrote: post #4

> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seem…

I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.

The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.

1. Its a security issue.

2. Publicly available sessions make it much worse

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#36

Post by sandeepkd »

aix1 wrote: post #31
yojo wrote: post #9

Earlier quoted context omitted.

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?

Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits

1. The down side is that it cannot be used across the clients even for the same user

2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#38

Post by crazylogger »

dboreham wrote: post #7

Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?

Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#39

Post by Cynddl »

> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.

I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.

Post reply on HN