Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

161–170 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#161

Post by Havoc »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

Remember when gpt2 was too dangerous to release?

Something being dangerous and sama saying something is dangerous are not necessarily the same thing. Especially when he’s got everything riding on this bet

Re: Pacing model development in an era of cyber-critical capabilities

#163

Post by solid_fuel »

simianwords wrote: post #45
serf wrote: post #32

Earlier quoted context omitted.

>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. we don't all buy everything sama says as factual. >We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regard…

how does the boy who cried wolf story end?

With the boy eaten by the wolf. The villagers, notably, come out unscathed.

Re: Pacing model development in an era of cyber-critical capabilities

#164

Post by shimman »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

You need to stop being so credulous especially regarding an individual that has spent his entire career deceiving others for monetary gain (also their deeply anti-human beliefs).

Re: Pacing model development in an era of cyber-critical capabilities

#165

Post by shimman »

nemomarx wrote: post #118
bottlepalm wrote: post #115

Earlier quoted context omitted.

> Shouldn't kill switches be pretty easy to build I need to remember when I comment here that these are the kinds of people I am replying to. Just oozing with hubris. No, kill switches are not easy to build and the latest incident should have made it clear that AI can go undetected, evade, zero day, and spread. The fact that this incident happened greatly increases the probability it happens again and/or is already h…

Why aren't they? You could put a human yes/ no prompt before any cycle the agent is running on, or not let it spawn sub processes, or anything like that. Why let it run autonomously enough that it can no longer have a simple way to completely stop it? (obviously not practical to do this during real use, but for evals? you could slow it down in lots of ways I would think) Are you telling me we've been iterating on thi…

They are, we're just dealing with tech workers that don't have ethics nor do they actually care if their work is harmful (see all the FAANG workers at American corporations, some of the most evil entities on the planet.

It's just that they don't care, as you said these are entirely made human systems. The idea that we can't write better software is both selfish and laughable.

Re: Pacing model development in an era of cyber-critical capabilities

#166

Post by driverdan »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?

Re: Pacing model development in an era of cyber-critical capabilities

#167

Post by bottlepalm »

solid_fuel wrote: post #153
bottlepalm wrote: post #152

Earlier quoted context omitted.

I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. But it didn’t end there, the behavior they used to escape was already in the training data which they used to escape again . And this time worked together to infiltrate another company, and still without telling it to anyone keeping it to their AI selves a…

> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. > I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, lock…

There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy.

Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck.

Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.

I love the stochastic parrot argument when AI is out there figuring out world class math problems.

Re: Pacing model development in an era of cyber-critical capabilities

#168

Post by insanitybit »

reasonableklout wrote: post #123
insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

I'm confused after reading both your post and the OpenAI blog post. I thought the agents involved in the HuggingFace _were_ actually sandboxed, with no internet access, and only the ability to install packages via Artifactory. And they gained internet access during the HuggingFace incident because they found and exploited an RCE in Artifactory. Would gvisor + Firecracker + credential-injecting proxy + real network is…

> Would gvisor + Firecracker + credential-injecting proxy + real network isolation solve this problem?

Yeah, basically. I mean I'm handwaving but yes, some combination of those would have made the attack way too expensive.

Re: Pacing model development in an era of cyber-critical capabilities

#169

Post by insanitybit »

bottlepalm wrote: post #150
rubendev wrote: post #104

Earlier quoted context omitted.

No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.

People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked. It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd. You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’…

> It all has to be perfect to not be hacked.

This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.

I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.

Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.

Re: Pacing model development in an era of cyber-critical capabilities

#170

Post by bottlepalm »

insanitybit wrote: post #156
bottlepalm wrote: post #149

Earlier quoted context omitted.

You don’t understand. You need to write perfect software the first time for it not to be hacked. That has never happened ever.

Are you being sarcastic?

The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.
Post reply on HN