Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be skeptical of OpenAI's rogue hacker agent story

theguardian.com

Post reply on HN
Print view View original

211–220 of 321 posts

Re: Be skeptical of OpenAI's rogue hacker agent story

#211

Post by munk-a »

johntb86 wrote: post #155
bluGill wrote: post #98

Earlier quoted context omitted.

I don't care about intent. That it happened is unacceptable. Ignorance is not an excuse

Intent often matters in the law (aside from certain laws with strict liability). You intentionally drive your car into someone you hate and kill them => murder, go straight to jail. You're driving along normally and someone who's chasing their pet cat suddenly runs into traffic and you hit them => no charges. Sometimes you can be charged with negligence for not taking enough care to prevent something, but then you ha…

If you're driving along normally and then tie a blindfold around your face and hit someone you've taken actions that still expose you to liability. And, in the case that you hit someone who is chasing their pet cat into traffic then you'd better hope you were following every facet of safe driving - being distracted, drunk or on your phone could easily result in you being charged.

Re: Be skeptical of OpenAI's rogue hacker agent story

#212

Post by [deleted] »

ctoth wrote: post #206
dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

Why do none of these few constrained ways to view this complex situation (nice gig if you can get it, agenda setting) include "and also this looks a heck of a lot like the stuff that the LW folks have been warning about for years and maybe we should slow down or stop ?" Because that was my takeaway.

[deleted]

Re: Be skeptical of OpenAI's rogue hacker agent story

#213

Post by snarkconjecture »

I think the most important question here is whether Huggingface was "in on it" – whether they knew in advance that this was going to happen and agreed to it. Second most important is whether OAI caused this to happen on purpose, with HF as an unwilling target.

Three options:

(1) Both companies are lying through their teeth and the entire thing is bullshit

(2) OpenAI hacked another company's servers and deliberately gambled on felony charges for the sake of PR

(3) OpenAI is telling the truth and a rogue AI agent hacked another company, opening them up to legal liability accidentally

All three of these are a lot more interesting than the "skeptical" story that this is just OAI doing business-as-usual PR hype, nothing to see here, move along.

(FWIW, I strongly doubt HF was in on it.)

Re: Be skeptical of OpenAI's rogue hacker agent story

#214

Post by kalkin »

why_at wrote: post #208
kalkin wrote: post #205

Earlier quoted context omitted.

Let's start by noting you made a factual claim that you apparently had no backing for whatsoever. I'd recommend against doing that. > they would be perfectly capable of keeping it just between themselves and not putting out press releases about it. In some jurisdictions there are legal requirements about disclosing cybersecurity incidents, and it's a well-established best practice even where there aren't. Beyond that…

Respectfully, calling this misinformation is unwarranted. The featured article is about exactly this. >On 14 February 2019, OpenAI announced a language model called GPT-2 >OpenAI declared GPT-2 was too risky to release, citing concerns about safety and abuse. >People with power and money took note: in July of that year, Microsoft invested $1bn in OpenAI. It's not a 100% gold standard RCT or whatever, but the idea tha…

> In 2016, the Chicago Cubs won the World Series

> This broke a hundred-year drought

> A few days later, Donald Trump won the 2016 presidential election

It's not a 100% gold standard RCT or whatever...

Re: Be skeptical of OpenAI's rogue hacker agent story

#215

Post by jasonfarnon »

why_at wrote: post #184
ACCount37 wrote: post #94

By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door. "It's a marketing stunt" is just denial trying to look like it's being clever.

I think you are conflating skepticism about AI companies' motives with skepticism about their capabilities. Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. The fact that this outcome is good for their business and stock price makes me suspicious about how much this was a complete accident vs an "accident" that they allowed to happen by setting up the right environmen…

"Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. "

So what? Under your scenario the frightening power of this model is still there. What's the relevance of openAI's marketing team being happy? The most you can hope for, if the facts of the exploit and intrusion are true, is that openAI was a little lax in its guards. And that isn't very reassuring. The scary thing about pistols are the power, having a safety on there doesn't make me relax much.

Re: Be skeptical of OpenAI's rogue hacker agent story

#216

Post by ACCount37 »

jackb4040 wrote: post #172
ACCount37 wrote: post #109

Earlier quoted context omitted.

Do you think that works? Just prompt a model "be good" and it stops doing anything bad? It never fucking worked that way and maybe never will. Prompts don't define model behavior. Prompts steer model behavior. Instruction-following over long horizons is NOT a guarantee in LLMs. Instructions doing what you want them to is NOT a guarantee in LLMs. Saying "don't exploit the box please pretty please" might actually cause…

There is a clear difference between saying not to do something because it's immoral, and saying doing that thing would be futile. In the Sopranos, there's an episode where a coffee shop protection racket is ruined because a local shop is replaced by a corporate chain that accounts for every cent daily, and immediately fires any employee involved in a discrepancy. In this case, the theft was prevented not by convincin…

Do you think "don't do this thing because you'll get zero evaluation score" is somehow perfectly reliable where "don't do this thing because it's bad" isn't?

Haha! Nope!

LLMs have consequentialist thinking sometimes. And sometimes they don't. Sometimes they follow the prompts, and sometimes they don't. There's NO single magic prompt that fixes all the weird behavior of modern LLMs, and it's baffling that anyone who has ever interacted with an LLM would expect there to be one.

Re: Be skeptical of OpenAI's rogue hacker agent story

#217

Post by dwoosley »

ctoth wrote: post #206
dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

Why do none of these few constrained ways to view this complex situation (nice gig if you can get it, agenda setting) include "and also this looks a heck of a lot like the stuff that the LW folks have been warning about for years and maybe we should slow down or stop ?" Because that was my takeaway.

That’s meant to be captured by point one with the model just being that advanced but more of a negative spin on it. If I felt option 1 was more likely, I think I’d have to agree with you there. Still, there currently are some gaps with that view in my opinion.

Re: Be skeptical of OpenAI's rogue hacker agent story

#218

Post by sfink »

bluGill wrote: post #11

I don't care how it happened someone should be arrested for illegal intrusion. Agents don't work on their own, someone is responsible. If nobody else the CEO for allowing something unsupervised. Hugging face also needs someone arrested for not providing security but that is a lesser charge.

I agree with the attacker side. The actions of autonomous agents are absolutely the responsibility of the one or more humans that enabled them to take that action. Whether that means someone is arrested, maybe or maybe not, but at least there should be a hefty fine.

I disagree with the defender side. It's not an unreasonable end state, but we're nowhere near there now. It would require holding company employees legally responsible for the security of their services, which means the risk of being employed as a (defensive) security professional is much higher, which means pay needs to be much higher and insurance needs to be available, etc. It's a very different world.

On the weekends, I'm coding up a list management app with a sync server. It's unreleased but exposed to the internet. (This is not hypothetical.) If that server ends up being used as part of an exploit chain, am I legally liable too?

Forget about age verification, now you want to associate every exposed port on the internet with a legally responsible human?

Re: Be skeptical of OpenAI's rogue hacker agent story

#219

Post by toddmorey »

dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

One way to think about it is that more powerful models mean solid best practices are more important than ever, so humans moving too quickly / carelessly bites us more than ever.

If a typical SaaS platform moved and shifted this fast with this many downstream consequences, we’d tell them to slow the fuck down, stop launching new features, and focus on security for a second.

But in AI I guess the idea is that more power and intelligence will solve for everything else.

Re: Be skeptical of OpenAI's rogue hacker agent story

#220

Post by devnonymous »

dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

> So then it’s seems it’s either that this was intentional(ish) or bad security

My take - don't attribute to intention that which can be sufficienty explained by inexplainability or incompetence (although I doubt the latter).

Post reply on HN