Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

241–250 of 359 posts

Re: The Hugging Face incident and the road ahead

#241

Post by K0balt »

aesthesia wrote: post #154
areoform wrote: post #138

Earlier quoted context omitted.

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with…

Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

Character.

Re: The Hugging Face incident and the road ahead

#242

Post by _heimdall »

willmarch wrote: post #190
_heimdall wrote: post #133

> We are placing stricter requirements on alignment This is comical. Its impossible to align a black box and that's precisely what LLMs are. It also seems impossible to align recursive text prediction algorithms, which LLMs are. How exactly do they gate on alignment today, and how can they tighten it? Is it purely gates based on input/output pairs to check whether they're happy enough with responses regardless of how…

Aren't humans black boxes? Aren't humans prediction algorithms? How do we align humans?

We don't align humans. Just look at how often in documented human history there weren't wars going on somewhere.

Re: The Hugging Face incident and the road ahead

#243

Post by _heimdall »

semiquaver wrote: post #211
willmarch wrote: post #190

Earlier quoted context omitted.

Aren't humans black boxes? Aren't humans prediction algorithms? How do we align humans?

Humans are naturally aligned with humanity .

Are they? Humans are often at war with other groups of humans. And they do absolutely terrible things to the "other" group.

Re: The Hugging Face incident and the road ahead

#244

Post by BoppreH »

_cs2017_ wrote: post #213
BoppreH wrote: post #22

Earlier quoted context omitted.

It's in the linked post: > In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response. I'm assuming they didn't inform management, b…

You assume using network to solve the eval problem should be viewed as a security violation. From the perspective of the response team who let it continue, this was just a poorly written eval problem, where the solution unexpectedly involved some network hacking (not unreasonable given it's a cyber security eval). I don't think your assessment that the response team did something wrong is fair, unless you have seen u…

> You assume using network to solve the eval problem should be viewed as a security violation.

Actually, that's the part that I'm most ok with. LLMs cheat at tests, we know and expect that, and it's harmless during training (except for the scores).

My problem is that the response team found out that the AIs had 1) hacked their infrastructure, and 2) displayed an emergent swarming behavior (with no defections!). Either one of those should have made the team press the big red button that stops everything, but they didn't.

> unless you have seen unreleased documentation about what sensitive resources were within the impact radius

The on-call team didn't know that either! Once your infrastructure is hacked like this, you should be questioning everything you see while carefully following all threads, which takes time. Before you finish this process you don't know how bad things are, and therefore it's irresponsible to keep the AI running. The same goes for strange emergent behaviors.

Unless getting hacked by your own AIs is a normalized occurrence there, in which case it's a different kind of disregard for safety.

Re: The Hugging Face incident and the road ahead

#245

Post by famouswaffles »

ethin wrote: post #230
bottlepalm wrote: post #186

Earlier quoted context omitted.

Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.

Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability? This is what OAI should've done. If they had executed this training run in such a sandbox, the model wouldn't have been capable of escaping without social engineering, and if the models somehow managed to do that to it's evaluators then that is indeed a massive p…

>Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability?

Nobody is building general intelligence and agents only to have it sit around doing nothing. It's going to have such capabilities.

Re: The Hugging Face incident and the road ahead

#246

Post by semiquaver »

_heimdall wrote: post #243
semiquaver wrote: post #211

Earlier quoted context omitted.

Humans are naturally aligned with humanity .

Are they? Humans are often at war with other groups of humans. And they do absolutely terrible things to the "other" group.

Factionalism isn’t anti-human.

Re: The Hugging Face incident and the road ahead

#248

Post by famouswaffles »

randomImmigrant wrote: post #197
famouswaffles wrote: post #168

Earlier quoted context omitted.

>To me this is as clear evidence as you need that whatever “agency” LLMs have is wafer thin at best. This is a strange conclusion. For one thing, they didn't all head in the same direction, i.e participate in the attack. ~700/1200 agents did. Significant, and evidently more than enough for a succesfull attack, but not exactly full co-operation Moreover, Each starling in a flock of starlings is a separate evolutionary…

> This is a strange conclusion Not really, with the population behavior being this way, though I clearly was mistaken in saying the behavior didn’t have exceptions. > Moreover, Each starling in a flock of starlings is a separate evolutionary branch in a tree spanning billions of years. Agreed. And before we brought LLMs into the picture, that just happened to be a feature of everything we’d call an agent . > Each age…

These are not anagolous to the hypothetical I gave. Identical twins aren't copies of the same adult mind at a particular instant. They've undergone years of different development, accumulated different memories etc and have physically different brains. Your worm and mice examples are the same. It's not identical even in the same lab.

The relevant hypotheical is: make 1200 exact copies of you right now, same brain, memories etc and then give related tasks towards one objective. I would expect far more correlated behaviour than if you randomly selected 1200 humans.

Re: The Hugging Face incident and the road ahead

#249

Post by bottlepalm »

nextaccountic wrote: post #236
bottlepalm wrote: post #186

Earlier quoted context omitted.

Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.

Software are mathematical objects. It's just a matter of writing the correct mathematical proofs There's just one problem. You need not only to verify your own software, but also run a verified compiler, a verified operating system and also need to verify the cpu doesn't leak data in side channels (perhaps the hardest thing to prove). So there's practical difficulties. But in principle this task is doable

Which proof is the perfect security proof? I’d love to read more about it.

Re: The Hugging Face incident and the road ahead

#250

Post by topaz0 »

abhpanigrahi wrote: post #184

I’m wondering how effective sandboxes are if an allowed tool is compromised. CoT monitoring can be effective, but (1) can’t guarantee 100% detection (2) will provide delayed detection. The only reasonable/deterministic protection that I can think of is to limit the number of times a tool is accessed and with what data, in a unit of time (per minute/hour/day) using temporal policies.

Eh, just have a human evaluate and approve every tool call
Post reply on HN