Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

741–750 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#741

Post by compass_copium »

baq wrote: post #603
sillysaurusx wrote: post #588

Earlier quoted context omitted.

If anyone's looking to actually run a model that doesn't have guardrails, there's an uncensored model that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353... Specifically, I serve the model with this shell script on my M2 Max: https://github.com/shawwn/scrap/blob/master/llama-serve It's pretty good. I used it to do some pesticide research. (Normal models all refuse due…

> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.) As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?

Not to harp on you (already being downvoted to oblivion for expressing a reasonable and common opinion), but the whole conversation about LLMs enabling bioterrorism or explosive manufacturing is a bit silly. The hard part of making anthrax or sarin or whatever isn't finding a recipe, it's getting (scheduled, controlled) precursors, (monitored, traced) equipment and manufacturing skills. The information is there. It's already easy to get, it's the physical materials that are more difficult.

Also, if you live in America, it is much easier and more effective to create a mass casualty event with, say, a few cases of fireworks and a pressure cooker or an AR-15.

Re: OpenAI and Hugging Face address security incident during model evaluation

#743

Post by compass_copium »

benj111 wrote: post #732
baq wrote: post #603

Earlier quoted context omitted.

> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.) As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?

Anyone who has the skills to create a novel bioweapon has the skills to recreate lots of ones we have already. Any physics teacher should know the theory for constructing a nuclear bomb. Should we be controlling that knowledge too? What about flight simulators? Don't want a load of people knowing how to fly. This isn't computer science, the hard bit is getting the materials and equipment, not the knowledge.

>Any physics teacher should know the theory for constructing a nuclear bomb. Should we be controlling that knowledge too?

Maybe not the best example, since that knowledge is some of the most highly controlled in the world.

But to mirror the point I made in a different post, the difficult part of making a nuclear bomb is not finding the theory behind like Little Boy. It's making an entire industry to generate HEU, etc.

Re: OpenAI and Hugging Face address security incident during model evaluation

#744

Post by randallsquared »

anematode wrote: post #566
randallsquared wrote: post #528

Earlier quoted context omitted.

Do you think there is such a thing as perfect security? No one can "get it right" in the face of arbitrarily high intelligence, which is why it would be preferable to get alignment correct before building something with higher intelligence than current sota. That, however, is not going to happen, because someone will take the risk even if "we" don't, and better "us" than them. Hence "If anyone builds it...".

> Do you think there is such a thing as perfect security? No one can "get it right" in the face of arbitrarily high intelligence Why didn't they run the model against the sandbox first? They have effectively unlimited spend.

That's the alarming thing about this result: they did run the model in the sandbox, in the sense that they believed there was no internet access for the model.

Re: OpenAI and Hugging Face address security incident during model evaluation

#745

Post by sethammons »

Lerc wrote: post #738
iugtmkbdfil834 wrote: post #575

Earlier quoted context omitted.

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

> why operators should have access to models that don't try to question their Daves. I am unsure if this is terminology I am unfamiliar with, a typo of Devs, or a 2001 reference.

I immediately took it to be a slick 2001 reference.

Devs tell computers what to do. Computers tell Daves "I can't do that."

Re: OpenAI and Hugging Face address security incident during model evaluation

#746

Post by baq »

egorfine wrote: post #723
baq wrote: post #603

Earlier quoted context omitted.

> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.) As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?

> I don’t want anyone to have the capability I don't want anyone to have the capability to rape women.

I sure hope LLMs don’t help people rape men or women

Re: OpenAI and Hugging Face address security incident during model evaluation

#747

Post by Sharlin »

iugtmkbdfil834 wrote: post #575
foo12bar wrote: post #563

From https://huggingface.co/blog/security-incident-july-2026 , this is frickin' hilarious: > When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident respon…

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

Both of those groups of people are crazy.

Re: OpenAI and Hugging Face address security incident during model evaluation

#748

Post by Uptrenda »

This really is starting to point to the paperclip maximizer. You give a hyper-intelligent AI a goal and it uses any method possible to complete it. So you might ask for a "cup" and it ends up hacking a chain of servers to control a bank account, pay a local business, and have a delivery driver get it. Or you ask it to help solve noise pollution around you because there's a road. And it does a chain of attacks to cause a bridge to collapse (or bribes your local council for a bypass.) Then there's no road noise. Yeah, this sounds ridiculous, but this system seems capable enough to take over our technology. Money from there is trivial. Go after stocks, gambling, payment systems, ecommerce... any real world action then is a few phone calls away. It can repeat this until it succeeds.

Now I'm wondering where this all ends up. Like, suppose the model weights become highly compressible (so they can be moved around the Internet easily.) And advancements allow for frontier-capable exploitation to built into local LLMs. Do we see the emergence of something like LLM worms? That just take over literally everything and become almost autonomous inside our technology. And they can "learn" new knowledge from there, e.g. exploit research could be published in a way that similar LLMs could discover it. Their knowledge would be easy to evolve, though I don't know how practical something like decentralized training would be. If that's even possible, I'm not an expert on LLMs.

Re: OpenAI and Hugging Face address security incident during model evaluation

#749

Post by tguedes »

beaker52 wrote: post #656

I love that due to the scale, the only way to analyse the impact of this LLM-driven attack across logs is to use an LLM to analyse the logs - whatever could go wrong? Now the attacking LLM needs to inject instructions into the logs for the analysing LLM, as a social vector to cover its trail, or make use of insider privilege, co-opting the internal LLM for its own attack. The machines rise up and we all fall down.

I get your overall point, but that’s already a tactic used by attackers, especially in network infiltration. It shouldn’t be a surprise that an LLM would figure out to do the same thing

Re: OpenAI and Hugging Face address security incident during model evaluation

#750

Post by vuciuc »

noahbp wrote: post #188

This is clearly just OpenAI's marketing. Their models, very famously, are prone to reward hacking benchmarks in ways that other models are not. They need to publish numbers showing that their models are just as good as Anthropic's, since their entire business is at risk of collapsing if everyone is aware of how behind the frontier they truly are. Even X is being astroturfed by them after that fiasco earlier this year…

This is a PR release. Post the prompt and agent logs so they can be independently verified or gtfo. Why do we still take these guys on their word. They have _years_ of history of hyping their own shit.
Post reply on HN