Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

191–200 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#191

Post by enraged_camel »

hirvi74 wrote: post #181
jimrandomh wrote: post #177

Earlier quoted context omitted.

The victim, Huggingface, told us. Or rather, they told the police first, setting up a situation where it was no longer possible for OpenAI to sweep it under the rug. Skepticism can be healthy, but you've got to follow up and actually check things. If you're skeptical unconditionally and don't check, you get tricked into being as skeptical of scandals as you should be of sales pitches.

I think the skepticism surrounding the Hugging Face attack is not about whether the attack actually happened, but whether it was truly accidental.

I see. Following your conjecture, there are two possibilities:

1. It wasn't an accident. OpenAI explicitly directed its agents to hack Hugging Face. Despite the fact that such a thing is a federal crime that carries prison sentence.

2. It wasn't an accident. OpenAI and HuggingFace conspired and let the hack happen for publicity.

Is there anything I'm leaving out?

Re: Pacing model development in an era of cyber-critical capabilities

#192

Post by XMPPwocky »

solid_fuel wrote: post #190
zmgsabst wrote: post #187

Earlier quoted context omitted.

> It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. These can both be true, particularly when there is substantial state associated with each token prediction.

> These can both be true, particularly when there is substantial state associated with each token prediction. The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point…

> The state is entirely internal to the network and disappears after a token is generated,

Yes and no, but mostly no, at least within a context window.

Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately).

But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse.

So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders).

> Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.

This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)

However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on.

One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do).

The brain being squishy doesn't make the thing easy to kill.

Re: Pacing model development in an era of cyber-critical capabilities

#193

Post by solid_fuel »

XMPPwocky wrote: post #192
solid_fuel wrote: post #190

Earlier quoted context omitted.

> These can both be true, particularly when there is substantial state associated with each token prediction. The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point…

> The state is entirely internal to the network and disappears after a token is generated, Yes and no, but mostly no, at least within a context window. Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately). But nobody actually implements this, because each token depends on stat…

> So, in practice, inference computes a very rich vector of state- at each layer, for each token.

And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context.

> This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)

Your brain continues to run without sensory input. LLMs do not.

Re: Pacing model development in an era of cyber-critical capabilities

#194

Post by scottmas »

insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

You literally just said a whole bunch of words that are literally gibberish to your average software developers. The devs at openai are good, but very few of them are cyber focused, so it’s not surprising IMO. And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.

But yes you are right it would have helped a ton.

Re: Pacing model development in an era of cyber-critical capabilities

#195

Post by XMPPwocky »

solid_fuel wrote: post #193
XMPPwocky wrote: post #192

Earlier quoted context omitted.

> The state is entirely internal to the network and disappears after a token is generated, Yes and no, but mostly no, at least within a context window. Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately). But nobody actually implements this, because each token depends on stat…

> So, in practice, inference computes a very rich vector of state- at each layer, for each token. And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context. > This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!) Your brain co…

> My point here is there is no continuous state that is not computed from the context.

Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.

Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) - what makes you feel differently (or am I misunderstanding you again?)

Re: Pacing model development in an era of cyber-critical capabilities

#196

Post by oblio »

bottlepalm wrote: post #170
insanitybit wrote: post #156

Earlier quoted context omitted.

Are you being sarcastic?

The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.

You've just been given examples of safer technologies. Especially with LLMs being all knowing code churning machines, you'd expect safer systems to be cheap to build.

Re: Pacing model development in an era of cyber-critical capabilities

#197

Post by solid_fuel »

XMPPwocky wrote: post #195
solid_fuel wrote: post #193

Earlier quoted context omitted.

> So, in practice, inference computes a very rich vector of state- at each layer, for each token. And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context. > This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!) Your brain co…

> My point here is there is no continuous state that is not computed from the context. Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error. Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) -…

> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.

Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLMs, which are strictly feed-forward and do not have internal loops. I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.

You could claim that the prompt and the context fill that role in an LLM, but I don't believe they are equivalent because the internal state in an LLM gets compressed down to a token which is then added back into the context, as compared to that state continuing to change within the network itself.

It's a little hard to explain, so I'm sorry if this seems like rambling.

But I believe it matters, and ties into running without sensory input, precisely because without sensory input you would in fact be perfectly capable of doing something. You would be capable of developing a desire and planning to achieve it without any prompting, without sight, without sound, etc. This is in stark contrast to LLMs, which will not do anything without a prompt.

An LLM may say complete the sentence "I am feeling ___" but it doesn't actually have feelings that exist without that prompt. There is no recurrent network where "bad", "good", "happy" might live before the query. It can't sit there, start to feel bad, and then seek a way out of its own volition.

That changes how dangerous something is because if a malicious prompt encourages an LLM to hack something, and you change the prompt, the "impulse" to hack something is gone. If you stop prompting it, it doesn't do anything at all. It just sits there. A living being will act on it's own, and that makes a huge difference in how dangerous something can be. It's the difference between a tool and an actual being.

---

To hone it a little further, if I took your brain out of your head and stuck it in a jar but kept it alive, it would probably make you angry. And if I then gave you power - like the ability to use the network - you may be motivated to use that power to attack me.

If I take an LLM and stick it in a jar... nothing. It's paused. It's awaiting a prompt. It's not secretly building plans to hack my pacemaker and make my heart explode.

Re: Pacing model development in an era of cyber-critical capabilities

#198

Post by protocolture »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

>People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades.

Theres Hyperbole and then theres whatever this is.

Re: Pacing model development in an era of cyber-critical capabilities

#199

Post by protocolture »

bottlepalm wrote: post #167
solid_fuel wrote: post #153

Earlier quoted context omitted.

> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. > I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, lock…

There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy. Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck. Breaking out? Not fantasy,…

>Breaking out?

It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it.

But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations.

Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people.

A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise.

What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.

Re: Pacing model development in an era of cyber-critical capabilities

#200

Post by protocolture »

simianwords wrote: post #68
bottlepalm wrote: post #64

Earlier quoted context omitted.

Is my concern getting you excited? My marketing must be working.

The discourse gets muddled because there’s a certain sect of loud people who still think all of this is hype and AI will just die down soon. There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.

>There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.

I would but, Londons under 1 mile of horse manure because that trend never stopped and theres no electricity anyway because Bitcoin is using it all. Good thing people getting scared about runaway trends are never wrong?

Post reply on HN