Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Reasoning Traces from Proprietary LLM APIs

stolen-thoughts.com

Post reply on HN
Print view View original

41–50 of 325 posts

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#41

Post by [deleted] »

azinman2 wrote: post #19
myworkaccount2 wrote: post #17

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.

[deleted]

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#42

Post by tanh »

So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#44

Post by aix1 »

sandeepkd wrote: post #36
aix1 wrote: post #31

Earlier quoted context omitted.

I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?

Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits 1. The down side is that it…

Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway?

> a per user key would have solved this issue for sure

It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#45

Post by simonw »

This is a neat attack against those encrypted reasoning blocks you get back from APIs like OpenAI and Gemini and Anthropic:

> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext.

Should be easy for them to fix though: switch up the encryption key so it only works with the API for each specific model, rather than being shared across all of their models.

And indeed, the paper says it's been fixed by all three providers (though no news on how they fixed it):

> All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#47

Post by dboreham »

x312 wrote: post #15
dboreham wrote: post #7

Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?

The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.

Hmm, ok. So the attack doesn't involve decrypting the payload, only getting the server to do so. Since a model will do that if you just ask, what's so special about the attack?

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#48

Post by flexagoon »

x312 wrote: post #14

Super cool that this works. I'm surprised these companies re-use the same encryption key across models! I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888

> I'm surprised these companies re-use the same encryption key across models

I assume switching the model in the middle of a conversation is intended behavior (very useful in coding agents, for example)

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#49

Post by kgeist »

iamcoder18 wrote: post #11

This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.

In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#50

Post by elzbardico »

azinman2 wrote: post #19
myworkaccount2 wrote: post #17

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.

Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers.

Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it.

Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests.

About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it.

Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.

Post reply on HN