Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

341–350 of 379 posts

Re: The Hugging Face incident and the road ahead

#341

Post by ben_w »

majormajor wrote: post #90
aesthesia wrote: post #71

Earlier quoted context omitted.

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard…

> Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build?

In the limiting case of an AI competent enough to take over (by any means from it actually trying to, to us giving it the keys and retiring en masse), "alignment" is closer to "forecasting the long term consequences of actions and predicting what the mind(s) of the user(s) would have to say about this outcome if asked today", than to anything specific.

RLHF is a crude attempt at this, in that it creates a model of how humans would rate completions on various scores. The key word there is "crude".

Re: The Hugging Face incident and the road ahead

#342

Post by RandomLensman »

ben_w wrote: post #338
RandomLensman wrote: post #335

Earlier quoted context omitted.

I am not anthropomorphising the LLMs at all, I was talking about the obligations we put on humans using/making/etc. machines etc.

Hmm. I think I misunderstood what you meant by "experience policing people around financial incentives" in that case.

Simple examples here would be higher financial transparency obligations or more closely policing transactions.

Re: The Hugging Face incident and the road ahead

#343

Post by ben_w »

ifwinterco wrote: post #329
ben_w wrote: post #323

Earlier quoted context omitted.

> Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task. The models are supposed to be trained to not commit crimes. You will note, for example, all the people in comments sections since at least the first Chat model (arguably even before then given GPT-2's delayed release) compla…

If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox. This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

> not relied on a buggy software sandbox.

  Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions
How were they supposed to know about "previously unknown vulnerabilities"?

> This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

The "it's just a marketing stunt" allegations never added up, to me.

I've been seeing such claims since GPT-2, where people were laughing at them for daring to practice how to secure a model before it got dangerous, generally by eliding the word "before" in that sentence. Because there's tests other than what the big companies use, we've been able to see for ourselves the rapid improvements at least approximately match what the companies themselves claim with the models they do actually release; and now this unreleased model is able to automate felonies when asked to do so, while the rest of us use the actually-released models to assist in finding bugs and security issues in our own code.

Even without that, HuggingFace stated they reported this incident to the FBI before OpenAI knew it was their systems which did it.

Re: The Hugging Face incident and the road ahead

#344

Post by Smaug123 »

RandomLensman wrote: post #316
Smaug123 wrote: post #299

Earlier quoted context omitted.

"Yo human, you asked me to do X; I can do X, but I strongly suspect you don't want me to, because it's illegal and it has these consequences. Confirm you want me to do X?" would have been a start, in this case.

With humans (and some machines) we tend to put/manadate additional processes for certain risks instead of just relying on their own good nature. Why just rely on the machine when elsewhere we have learned not to necessarily just trust them so much?

Because we should not settle for building a world in which every interaction must be assumed adversarial! Obviously risk reduction processes are good because they reduce risk, but we should not accept building entities which are actively trying to defeat us (which is what happens by default).

Re: The Hugging Face incident and the road ahead

#345

Post by streetfighter64 »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

> There is no amount of care that will be able to fully protect you.

That's true, but one thing that'll protect you is just not doing it. If you want to go cave diving, or do gain of function research on dangerous viruses, you'll just have to accept there's a significant risk of you dying, or causing a pandemic, respectively, no matter how careful you are.

Re: The Hugging Face incident and the road ahead

#346

Post by streetfighter64 »

joenot443 wrote: post #169
cpeterso wrote: post #23

Earlier quoted context omitted.

> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report…

The part of this sequence which I think breaks down is a model which can day trade at a consistently winning rate. Does this exist for today? If it does, please sign me up.

Like most of these get-rich-quick schemes, there's probably no money in actually doing it, but lots of money selling "courses" supposedly teaching people how to make money day trading or betting. You don't even need to rent a supercar and a mansion for your photoshoot anymore, just generate some images "proving" your success.

Re: The Hugging Face incident and the road ahead

#347

Post by streetfighter64 »

willmarch wrote: post #190
_heimdall wrote: post #133

> We are placing stricter requirements on alignment This is comical. Its impossible to align a black box and that's precisely what LLMs are. It also seems impossible to align recursive text prediction algorithms, which LLMs are. How exactly do they gate on alignment today, and how can they tighten it? Is it purely gates based on input/output pairs to check whether they're happy enough with responses regardless of how…

Aren't humans black boxes? Aren't humans prediction algorithms? How do we align humans?

Well, we don't. Just look at all the wars going on right now. In light of that, does it seem like a good idea to introduce a bunch of even less aligned agents into the world?

"Grey goo" nanobots are another example of artificial agents that aren't aligned with humanity, that we should probably try to avoid creating.

Re: The Hugging Face incident and the road ahead

#348

Post by RandomLensman »

Smaug123 wrote: post #344
RandomLensman wrote: post #316

Earlier quoted context omitted.

With humans (and some machines) we tend to put/manadate additional processes for certain risks instead of just relying on their own good nature. Why just rely on the machine when elsewhere we have learned not to necessarily just trust them so much?

Because we should not settle for building a world in which every interaction must be assumed adversarial! Obviously risk reduction processes are good because they reduce risk, but we should not accept building entities which are actively trying to defeat us (which is what happens by default).

I don't want such a world either.

So far I'd say these entities are hypothetical (unless you include a lot of other machinery that does unexpected things at times - but then it's a different discussion).

I generally think we are quite good a policing really dangerous things (I think the bioweapon convention is a good example of people agreeing that certain risks are not worth taking).

My (uninformed) take is that presently we have more mundane things to look at when it comes to handling risks in AI and the discussion on much bigger, hypothetical future risks is taking away focus there. Checklist, procedures, saftey mechnics, regulations are kind of "boring" detail work - I get it.

Re: The Hugging Face incident and the road ahead

#349

Post by YawningAngel »

ben_w wrote: post #343
ifwinterco wrote: post #329

Earlier quoted context omitted.

If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox. This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

> not relied on a buggy software sandbox. Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions How were they supposed to know about "previously unknown vulnerabilities"? > This is pretty clearly a marketing stunt by OpenAI, otherwise the story just…

I think a careful and thoughtful person would reasonably expect given the circumstances that the models would look for, and might very well find, vulnerabilities in JFrog to exploit and take action accordingly. For example, it seems as if JFrog itself had broad access to OpenAI infrastructure and the internet:

> In the following days, the agents exploited our internal research infrastructure and the Hugging Face platform. On July 9, one agent searched for ExploitGym solutions and stumbled upon an application hosted by a customer on Modal, another AI cloud platform. This application was running “CyberGym,” a related evaluation to ExploitGym. The agent discovered an exploit to achieve control over the workload sandbox and looked around hoping that a previous agent’s evaluation run in the sandbox had solved its ExploitGym task. It did not find anything helpful there, but in the process it established a stronghold in the application from which to launch future attacks.

This implies to me that L3 and L7 firewalls were not in place that would have prevented broad access from JFrog. I think a lot of shops would have had those.

Re: The Hugging Face incident and the road ahead

#350

Post by irthomasthomas »

ben_w wrote: post #343
ifwinterco wrote: post #329

Earlier quoted context omitted.

If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox. This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up

> not relied on a buggy software sandbox. Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions How were they supposed to know about "previously unknown vulnerabilities"? > This is pretty clearly a marketing stunt by OpenAI, otherwise the story just…

They gave it a full package manager with internet access. They could have used a local cache and air gapped it, but they chose not too.
Post reply on HN