Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

781–790 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#781

Post by iugtmkbdfil834 »

insanitybit wrote: post #776
friendzis wrote: post #609

Earlier quoted context omitted.

"if only we could align the models just a tiny lil bit better" is a rehashed "if only we could escape untrusted inputs just a tiny lil bit better" from 2000s, that were RIPE with various form of malicious injection. Every command+data channel in existence has been and will continue to be exploited one way or another, because the solution space is for all intents and purposes unbounded. Sure, highly defensive escaping…

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

But see... this is why it is a perfect long term job in the age of AI;p Them peoples think they found ultimate hack.

Re: OpenAI and Hugging Face address security incident during model evaluation

#783

Post by fatata123 »

scoring1774 wrote: post #221

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. It's remarkable that building a society based around having to do something so you can go do your hobbies at home…

[dead]

Re: OpenAI and Hugging Face address security incident during model evaluation

#784

Post by sccvcxv »

scoring1774 wrote: post #221

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. It's remarkable that building a society based around having to do something so you can go do your hobbies at home…

[flagged]

Re: OpenAI and Hugging Face address security incident during model evaluation

#786

Post by sccvcxv »

superb_dev wrote: post #264
noahbp wrote: post #188

This is clearly just OpenAI's marketing. Their models, very famously, are prone to reward hacking benchmarks in ways that other models are not. They need to publish numbers showing that their models are just as good as Anthropic's, since their entire business is at risk of collapsing if everyone is aware of how behind the frontier they truly are. Even X is being astroturfed by them after that fiasco earlier this year…

Is OpenAI truly behind? Just anecdotally I recently fully switched to using Codex at work because it feels a lot more competent

man they burned crazy amounts of money on stupid irrelevant stuff

they are in deep trouble and its all their own fault.

Re: OpenAI and Hugging Face address security incident during model evaluation

#787

Post by lyu07282 »

nananana9 wrote: post #724
eternauta3k wrote: post #700

Earlier quoted context omitted.

I wonder how these companies airgap the weights while allowing prompts to come in and outputs to come out.

You pretty much get this for free. The box doing the inference has one job, to do inference - there's no reason for it to interpret the inputs/outputs in any way, so as long as it doesn't do something stupid like modify the output stream and echo out the model weights, you're fine. Content filtering, interpreting tool calls, etc can all happen downstream on boxes that don't have access to the weights.

I would imagine a hypothetical super intelligence could manipulate its own runtime behavior to exploit bugs in the GPUs itself, like think rowhammer like attacks on the gddr memory of GPUs or against firmware or against Infiniband/NVlink. Everything above that runs so much code full of bugs on CPUs full of bugs on storage with firmware full of bugs.

You would have to completely airgap the entire cluster and put it in a faraday cage, the people working on that would have to physically be in the datacenter and burn CDs to one-way transfer data over. Like in all the hypothetical ASI scifi scenarios they always assumed that's a given, they thought it obvious we would put actual "effort" into sandboxing the AI, so they talked a lot about how AI would use social engineering attacks to convince humans to help it escape it's sophisticated sandbox anyway. Turns out they were all wrong about that part, it won't even be necessary.

Re: OpenAI and Hugging Face address security incident during model evaluation

#788

Post by gallerdude »

andai wrote: post #590

> This incident occurred during an internal evaluation which prompts models [with safeguards disabled for evaluation purposes] to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. Researcher: hack me Model: understood Researcher: oh my god

But it wasn't explicitly told to hack HuggingFace. It was told "answer this security question", and it's answer was to break into the teacher's desk to find the answer key.
Post reply on HN