Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

821–830 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#821

Why don't we just have a pause, while we think about the consequences? Stop release of the latest generation of models while society develops to a level where we can deal with it?

Trust the invisible hand of the market to sort this out. Involving society in anything the market does is only communism in another guise.

But in all honesty, first try to convince the investors that a pause would be good. They only care about their money and society is an annoyance that regulates their ability to make even more money.

Whatever money wants, money gets.

Re: OpenAI and Hugging Face address security incident during model evaluation

#822

Earlier quoted context omitted.

"if only we could align the models just a tiny lil bit better" is a rehashed "if only we could escape untrusted inputs just a tiny lil bit better" from 2000s, that were RIPE with various form of malicious injection. Every command+data channel in existence has been and will continue to be exploited one way or another, because the solution space is for all intents and purposes unbounded. Sure, highly defensive escaping…

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

No, that's a misconception that led to decades of vulnerabilities.

You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done.

Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs.

If your hand rolled implementation just chains `.replaceAll("", input)` and `.replaceAll("", input)` the escaping result depends on evaluation order, at least on already pre-escaped inputs if your instruction sequences are single-element. Even if you get it right and don't reduce escaped sequences to double escape + unescaped, you are still dealing with stray escape sequences: "John o\'Doe".

You have to meticulously (all the way through your call tree and even through persistent storage cycle!) track if a value has already been escaped and whether it needs escaping. As paradoxical as it may sound, meticulous and defensive escaping produces tons of bugs. If the project decides to escape raw inputs right when passed in, escaping them once more before passing out (to storage, another process) is a bug that you cannot easily statically test against.

On top of that, various modules that you interact with (storage, libraries, modules pulled from another team) will have different escaping semantics: some will apply escaping on their own, some will expect input to be "sanitized" and treat it raw. The semantics can even be different on different paths: write to storage module accepts input as is, but retrieval method "helpfully" runs escaping.

Furthermore, in different contexts the escaping rules are going to be different. In a web world, what's safe to write directly to html, pass to js `alert`, and pass to sql query are entirely different things. Input safe to dump into html is not necessarily safe to pass to string-interpolating SQL DTO layer, and vice versa. Then the DBA changes config to allow variables in queries and your escaping _semantics_ are now entirely different.

It's a minefield with essentially unbounded surface. You will trip up. People have tried to solve the problem for decades. Very smart people have tried. They all have failed.

Re: OpenAI and Hugging Face address security incident during model evaluation

#823

Earlier quoted context omitted.

And if you take it at face value, then they are more or less saying that they kinda are close to not being able to control at all the thing they developed, which is pretty crazy too.

We are not going to know we have crossed the line until it's been crossed and we can look back and say "Oops. We should have done something back then."

People have been sounding that alarm for well over a year now, just not anyone who has the power to do anything about it.

Re: OpenAI and Hugging Face address security incident during model evaluation

#824

All the people saying that this is pure marketing: Do you think that they are literally lying about what happened, or do you just think that what happened doesn't matter in any sense whatsoever, and that therefore the only reason they are telling people about it is a marketing purpose?

You know that people can plan the whole theatre?

I'm not quite sure what you mean by that, but it sounds like you're suggesting the lying about it.

Re: OpenAI and Hugging Face address security incident during model evaluation

#825
post #444

If you are attempting to run exercises like this, it is wildly negligent to not be running it in a physically-airgapped environment (potentially with a physical power shutdown). You can not tell me that OpenAI doesn’t have the resources or ability to run tests like this in a physically-non-networked environment w/ sufficient compute for its needs.

> wildly negligent to not be running it in a physically-airgapped environment Why should it be physically airgapped? Clients won't be doing that.

Because by the time it goes out to clients it (should be) thoroughly tested and aligned for safety, but at the present moment it isn't.

Re: OpenAI and Hugging Face address security incident during model evaluation

#826
post #797
post #600

Earlier quoted context omitted.

IMO they hope to make AI a strongly regulated industry, with OpenAI (and Anthropic) becoming military suppliers with their stronger models, and everything Chinese or open-weight gets banned. The competition from the open models is so strong now that this seems to be the only way to keep both companies afloat, given their dire financials. OpenAI probably hoped that they can achieve market lead and then lower the train…

thats economic suicide for the whole country. europe and china will never agree to rules that are obviously designed to put them in a permanent bad position. these regulations can only pass in america and nowhere else. if it doesnt end in a revolution then the united states will be the first ever 5th world country. openai and anthropic will stop any real innovation and focus on extracting profits from a failing econo…

[dead]

Re: OpenAI and Hugging Face address security incident during model evaluation

#827

Can someone not super-AI-pilled explain to a reasonable lay person why this matters? It seems like the comments here are a mix of: * The test was irresponsibly designed and protected * The model was particularly persistent in finding a way to access the network and exploit vulnerabilities * The model 'shouldn't' have done this But as far as I can tell: * The model didn't destroy anything on the way - it just was 'pap…

> it just was 'paperclip maximizing' to literally exploit

Why “just”? Paperclip maximizing is exactly one of the nightmare scenarios.

I’m not sure why you take comfort in knowing that it was just that.

Re: OpenAI and Hugging Face address security incident during model evaluation

#828

Earlier quoted context omitted.

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

No, that's a misconception that led to decades of vulnerabilities. You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done. Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs. If your hand rolled implementation just chains `.replaceA…

You can solve this with refinement types. It's entirely tractable in many systems in principle - APIs can be written to be safe, you can track data as separate from code, etc. None of this exists in an agent - you can't separate data from code, it's intractable because it's impossible in principle.

Re: OpenAI and Hugging Face address security incident during model evaluation

#829
post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

[dead]

Re: OpenAI and Hugging Face address security incident during model evaluation

#830
post #797
post #600

Earlier quoted context omitted.

IMO they hope to make AI a strongly regulated industry, with OpenAI (and Anthropic) becoming military suppliers with their stronger models, and everything Chinese or open-weight gets banned. The competition from the open models is so strong now that this seems to be the only way to keep both companies afloat, given their dire financials. OpenAI probably hoped that they can achieve market lead and then lower the train…

thats economic suicide for the whole country. europe and china will never agree to rules that are obviously designed to put them in a permanent bad position. these regulations can only pass in america and nowhere else. if it doesnt end in a revolution then the united states will be the first ever 5th world country. openai and anthropic will stop any real innovation and focus on extracting profits from a failing econo…

Europe would absolutely be stupid and servile enough to agree to this, unfortunately.
Post reply on HN