Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Reasoning Traces from Proprietary LLM APIs

stolen-thoughts.com

Post reply on HN
Print view View original

81–90 of 325 posts

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#81

Post by thefourthchime »

vinaigrette wrote: post #33

I must say right of the bat this is the best research paper/working paper in regards to its styling. Beautiful

I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.

This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#82

Post by SaltyBackendGuy »

supriyo-biswas wrote: post #79
myworkaccount2 wrote: post #27

Earlier quoted context omitted.

There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves? This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation. If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid…

It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.

Or even my fable credits run out mid task and need to switch back to opus >.<

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#83

Post by dannyw »

yubblegum wrote: post #30
x312 wrote: post #14

Super cool that this works. I'm surprised these companies re-use the same encryption key across models! I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888

Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI. (Thanks for the link. That’s an interesting idea!)

The provider has the hidden text anyway; this isn’t customer managed encryption.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#84

Post by dannyw »

nervai wrote: post #8

Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though. A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267

Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#85

Post by dgellow »

andai wrote: post #74

If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.

Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#86

Post by HarHarVeryFunny »

dgellow wrote: post #85
andai wrote: post #74

If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.

Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches

For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#87

Post by dannyw »

SwellJoe wrote: post #26

"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world. And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.

Yeah, it’s also useful for prompt tuning, debugging and understanding how a model interprets your prompt.

Also really good for identifying any contradictions in your system prompt and context.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#88

Post by Terr_ »

niemandhier wrote: post #59

You cannot steal what is not owned. At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.

While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.

For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#89

Post by otterley »

Zambyte wrote: post #72
niemandhier wrote: post #59

You cannot steal what is not owned. At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.

Even copyrighted information can never be "stolen". It can only copied without authorization.

Stealing is not a word that applies only to physical objects.

Re: Stealing Reasoning Traces from Proprietary LLM APIs

#90

Post by lukewarm707 »

iamcoder18 wrote: post #11

This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.

their gpt-oss models do the same. i don't use closed models so i never thought much about it.
Post reply on HN