Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

871–880 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#871

Post by sandeepkd »

pixl97 wrote: post #769
sandeepkd wrote: post #515

Earlier quoted context omitted.

>That capability alone could hack half the US. This almost seems like believing in magic. What really has happened is you have collected all the hacking/abuse/malicious flows/code in one place. Greedy or A* algorithms have been discovered a long ago, the script is executing the flows for all possible permutations. Something has to be insecure to be hacked in the first place.

Looking at the hundreds of linux and windows CVEs in the past month tell me I have little to worry about things being secure.

Security exists in multiple layers. CVEs do not make a system vulnerable by default, there has to be a way to exploit them. Questions one can ask, is your system connected to internet, what does your fire wall looks like, which ports are open and why, what processes are running and why.

Most sensitive systems in the past had a concept of DMZ where one cannot reach through internet for security reasons except for very tightly controlled path. One had to go in the physical data centers for certain operations.

If you are running a script in a loop on your system which gets executes any/all commands from a third party system then you probably do not take security that seriously in the first place.

Re: OpenAI and Hugging Face address security incident during model evaluation

#872

Post by pixl97 »

jmalicki wrote: post #429
jmalicki wrote: post #155

Earlier quoted context omitted.

> This is science fiction, these models don't have access to their own weights The weights plus the architecture is the model. What do you even think "the model" or "the weights" are? The weights aren't some far off training concept, every time you type something into ChatGPT it's making a forward pass over the weights. It's as silly as saying "Computer programs don't have access to their binary compiled code at exec…

Oh yes, the agent won't have access to the weights via tool calls. But nothing would inherently stop an RLVR trained model from distilling a version of itself and proving it could regenerate that at runtime, if somehow it got off on an evil tangent and "decided to do so", much like the model hacked to get at the answers here, or the agent can hack out a sandbox to achieve its goals. It would be extremely impressive f…

I mean the agents just hacked an external system, so what keeps it from hacking the model hosting servers?

Re: OpenAI and Hugging Face address security incident during model evaluation

#873

Post by lukewarm707 »

TSiege wrote: post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

hugging face should not have bent the knee to openai for cyber access, they should have opened communication with openai by serving the c suite with a lawsuit. the courts should throw the book at openai, but they won't.

they are becoming untouchable. in terms of piracy, monopolistic activities, and now hacking competitors and exfiltrating their confidential data.

Re: OpenAI and Hugging Face address security incident during model evaluation

#874

Post by tomjen3 »

atwrk wrote: post #600
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

IMO they hope to make AI a strongly regulated industry, with OpenAI (and Anthropic) becoming military suppliers with their stronger models, and everything Chinese or open-weight gets banned. The competition from the open models is so strong now that this seems to be the only way to keep both companies afloat, given their dire financials. OpenAI probably hoped that they can achieve market lead and then lower the train…

I think that is probably too conspiratorial, if only for the reason that Europe is not gonna go along with it.

I work in tech in Europe and we have a fair number of customers who arelike. we can accept AI, but they must keep the data in Europe. That's trivial with an open weight. We literally cannot do it with Fable.

Re: OpenAI and Hugging Face address security incident during model evaluation

#875

Post by qlte »

faxmeyourcode wrote: post #394
rcr-anti wrote: post #110

At release the 5.6 Sol card noted substantially higher rates of actions 'a reasonable user would likely not anticipate and strongly object to'. METR made a post, https://metr.org/blog/2026-06-26-gpt-5-6-sol/ , that 5.6 Sol was "cheating", their word, so hard in long horizon benching it effectively couldn't be benchmarked. I wonder, is it this persistent and aggressive in all tasks or is this specific to benchmarks? A…

I've definitely noticed 5.6 sol being extremely trigger happy in ways other models, even 5.5, we're not. I would definitely categorize a few small incidents at work where it performed "actions a reasonable user would likely not anticipate and strongly object to." Just my anecdotal experience. For example discussing driver upgrade and subsequent password rotation and it didn't stop and ask me if I wanted to restart th…

I’m still using 5.5 and had it do almost exactly that same example on a task yesterday so doesn’t seem like a clear cut 5.5 vs 5.6 thing. It’s pretty trigger happy already once it gets any kind of “go” without specific restrictions.

Re: OpenAI and Hugging Face address security incident during model evaluation

#876

Post by ethin »

baq wrote: post #603
sillysaurusx wrote: post #588

Earlier quoted context omitted.

If anyone's looking to actually run a model that doesn't have guardrails, there's an uncensored model that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353... Specifically, I serve the model with this shell script on my M2 Max: https://github.com/shawwn/scrap/blob/master/llama-serve It's pretty good. I used it to do some pesticide research. (Normal models all refuse due…

> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.) As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?

This is nonsense. By this logic some random corporation should have total control over your computer and the inputs you feed it and the outputs it produces to ensure nobody who isn't "verified crazy" uses it. That's essentially what your saying.

These models are, ultimately, tools. I would never trust some random corporation (particularly one with a profit motive and hypocritical stance, which includes both OpenAI and Anthropic, to be clear) to decide what isn't and is considered "crazy" and who and who isn't "verified" not to be "crazy". Especially when these companies have time and time again demonstrated (1) that they cry wolf way too much which leads to nobody taking their claims about how "dangerous" their models are seriously and (2) incidents like this where OpenAI makes a claim ("Look at how dangerous our models are!") and then doesn't be smart and just... Slow the fuck down (and when testing these things, actually sandbox them properly, which obviously wasn't done here or this attack wouldn't have been even possible).

Re: OpenAI and Hugging Face address security incident during model evaluation

#878

Post by embedding-shape »

scoring1774 wrote: post #221

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. It's remarkable that building a society based around having to do something so you can go do your hobbies at home…

> Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal.

This happens from time to time when you work on optimizations and similar things, with less "smart" LLMs and under-specify what exactly you're out after. Asking them to make functions faster without clearly specifying what the function has to do, is a great way to replicate this too. Doesn't seem to happen as often with SOTA models though.

I think the early example of "I asked it to make the test suite pass, so it changed all the assertions" is pretty much the same variant of this, where it technically does what it is asked to do, yet in "clearly" (to humans) wrong ways.

Re: OpenAI and Hugging Face address security incident during model evaluation

#879

Post by ethin »

foo12bar wrote: post #587
iugtmkbdfil834 wrote: post #575

Earlier quoted context omitted.

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

And the fact they used a Chinese model, because none of the frontier models from very highly valuated top US companies support their very common and essential use case.

I also thought this was hilarious. The very "safety" measures these models have prevented them from doing... Something that is designed to increase safety?

Re: OpenAI and Hugging Face address security incident during model evaluation

#880

Post by butlike »

insanitybit wrote: post #776
friendzis wrote: post #609

Earlier quoted context omitted.

"if only we could align the models just a tiny lil bit better" is a rehashed "if only we could escape untrusted inputs just a tiny lil bit better" from 2000s, that were RIPE with various form of malicious injection. Every command+data channel in existence has been and will continue to be exploited one way or another, because the solution space is for all intents and purposes unbounded. Sure, highly defensive escaping…

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

Alignment in LLM Land is probably like 'laws' in Human Land. They exist, and you probably SHOULD follow them, but you don't HAVE to.
Post reply on HN