Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

161–170 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#161

Post by noman-land »

NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking.

AI can't do shit if you terminate the process or unplug the box.

The news here is that these researchers are irresponsible.

They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.

Re: Responding to the next frontier of critical cyber capabilities

#162

Post by miohtama »

noman-land wrote: post #161
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days . Found out it hacked and kept hacking. AI can't do shit if you terminate the process or unplug the box. The news here is that these researchers are irresponsible. They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they…

Also, every Linux comes with a firewall

Re: Responding to the next frontier of critical cyber capabilities

#164

Post by the8472 »

bottlepalm wrote: post #152
somesortofthing wrote: post #129

I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."

In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option".

And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.

Re: Responding to the next frontier of critical cyber capabilities

#166

Post by jackb4040 »

janalsncm wrote: post #156
jackb4040 wrote: post #147

Earlier quoted context omitted.

Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero. I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary…

> Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.

Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors.

It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of that scene from Terminator.

Re: Responding to the next frontier of critical cyber capabilities

#167

Post by drdexebtjl »

Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

I was replacing the factory OS from a NAS and asked Sol to do an inventory on the running system over SSH, so that I could replicate any hardware compatibility quirks and whatnot.

It reverse engineered a binary daemon that set fan curves and told me how I should set them up in the new OS. I didn’t ask for this, and I didn’t have reverse engineering tools installed. It just figured out it could run them using Nix.

The most worrying part, to me, is that it did it like it was nothing. It simply said “usr/local/bin/some-daemon sets the following fan curves”. I had to ask how it reached that conclusion for it to tell me casually that it had just read it straight from the x86_64 assembly.

No access to the source code is no longer a meaningful obstacle to these models.

Re: Responding to the next frontier of critical cyber capabilities

#168

Post by bottlepalm »

the8472 wrote: post #164
bottlepalm wrote: post #152

Earlier quoted context omitted.

In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option". And the thing they're already doin…

None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense.

He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans.

It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved.

The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.

Re: Responding to the next frontier of critical cyber capabilities

#169

Post by ywain »

tcp_handshaker wrote: post #124
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania. - Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity. - I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems I…

Normal rules do not apply because these companies are worth billions and powerful people don’t want them to fail. It’s neither specific to AI nor a new phenomenon.

Re: Responding to the next frontier of critical cyber capabilities

#170

Post by bobby-cb »

NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally.

I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

Post reply on HN