This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return.
Anyway, can someone explain the part about K3? What are they trying to say?
Stealing Reasoning Traces from Proprietary LLM APIs
61–70 of 325 posts
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#62[deleted]
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#63Earlier quoted context omitted.
The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.
Hmm, ok. So the attack doesn't involve decrypting the payload, only getting the server to do so. Since a model will do that if you just ask, what's so special about the attack?
The large models whose thinking traces are useful are safeguarded against this reasoning replaying. the small models are just designed for speed and efficiency, so these safeguards are a lot meaker, making the attack possible
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#64Earlier quoted context omitted.
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#65You cannot steal what is not owned. At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
[dead]
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#66This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return. Anyway, can someone explain the part about K3? What are they trying to say?
The interesting part is what they try to not to say: More indications for K3 is based on distillation from Claude and GPT.
From [1]:
> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.
> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s
> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.
[1] https://x.com/kotekjedi_ml/status/2087147042888114428?s=42
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#67Earlier quoted context omitted.
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
100% guaranteed that this research just forced this to happen now. Sucks.
or add some metadata and don't allow downgrading.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#68You cannot steal what is not owned. At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
they also can't claim to be the owner of the output either because they can't claim to have had a valid license for the inputs.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#69This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return. Anyway, can someone explain the part about K3? What are they trying to say?
Security is solved, but business needs overrides it
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#70This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
Muse clearly does it to some extent. Saw a lot of that running Glimmer locally.