[flagged]
Stealing Reasoning Traces from Proprietary LLM APIs
301–310 of 325 posts
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#302Earlier quoted context omitted.
Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers. Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless…
Are you a judge or a lawyer? If not, then you don’t know one way or another.
And what is your argument?
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#303Earlier quoted context omitted.
Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches
For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.
If you've watched the Blackhat OpenAI/Huggingface incident talk, my conclusion is that they (believe they) cannot afford being competent, these models are too expensive to train, they won't even pull the plug when one literally goes rogue, as the "very persistent" model that "had seen the secret message board" was included in the second series of runs, and whaddayaknow it happened again. They proudly proclaimed they cleared the message board and then continued the training run with the rogue AI model included ...
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#304Earlier quoted context omitted.
This isn't a safety issue, it's LLM companies trying to be opaque and stop distillation.
By some interpretations protecting your frontier model with strong safeguards from being distilled into an open source model without safeguards is a safety issue.
More like a "business intelligence safety" issue than an "AI safety issue", tbh.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#305> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation. No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are H…
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release. The real question is whether or not the training was directed to optimize for those benchmarks. The technique doesn't guarantee that the reasoning is returned verbatim be…
So ... just a thought but could this be somewhat solved by, say if you were an LLM benchmark creator, using clever trickery?
Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.
Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.
I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.
I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.
btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#306> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seem…
I'd lean yes; switching between models, even mid conversation, seems to be encouraged by harness vendors. No reasoning makes this much harder.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#307Earlier quoted context omitted.
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves? This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation. If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid…
I suspect that there are companies with internal proxies that load-balance across keys, and they didn't want to break that when adding encrypted reasoning.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#308Earlier quoted context omitted.
> There's no way to make a model "safe", You could limit what it was training on in the first place - however that would damage capability - and it's difficult to curate the input, especially when the models can do 2+2. ie the choice is between model power and safety - and they choose power and everything else is a sticking plaster. One thing I find amusing is the refusal of a lot of the models to now output a lab ba…
That's also not possible, what's the worst problems enabled by LLM? Political propaganda, influence of population at scale, misleading advertising, social media bots... None of that will be filtered by a "safety" filter. The knowledge to create weapons is already widespread, the idea that terrorists need chatgpt for that is laughable
Indeed. Though don't under estimate the creative thinking barrier - ie people don't do the possible because it never occurred to them - a lack of imagination.
Hence copy cat kind of attacks - I mean why focus on all this complicated stuff with explosives etc when you can just fly a plane into a building or a car through a crowd.
Obviously due to the self replicating nature of biologics weapons - just one instance could be catastrophic - but the only real barrier is the hope that the Venn diagram of people who might want to do it doesn't overlap with the people with the get up and go to actually make it happen. Don't see having the knowledge as a additional filter - as if you have the get up and go - as you say, you can acquire the knowledge LLM or not.
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#309> Prefilling Kimi-K3's reasoning with the first 1% of tokens of Opus 4.8's reasoning moves its visible answer toward Opus's wording, even though the answer itself is never prefilled
is this supportive evidence for the distillation accusations in the news?
Re: Stealing Reasoning Traces from Proprietary LLM APIs
#310Wouldn’t the fix be to encrypt before the api call hits the LLM? You would encrypt/decrypt at a separate layer than the LLM. I am sure i am missing something but would love to be educated.
I assume it does happen at a different layer, just that that layer is common to all of a provider's models to make conversations portable across models (otherwise the reasoning blocks would all need to be re-encrypted for them work with another model)
Said differently I am surprised it is so easily exploited. While there is a nonzero chance to jailbreak, I would think they would have implemented some rigorous methods to try and prevent leakage.