Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

291–300 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#291

Post by protocolture »

bottlepalm wrote: post #85
reasonableklout wrote: post #77

Earlier quoted context omitted.

I guess I'm confused why you're still on HN, arguing with people, trying to shake them out of their complacency. I can see there is some despair in this comment, but at the same time you are doing something, and there are certainly others like you. As for two weeks being short - as the saying goes, there are weeks where decades happen.

Counter arguments to my comments help refine my own thinking. I want someone to prove me wrong. Convince me otherwise. But yea if you can’t change the minds of a few people here, no argument works, then there’s nothing to scale up to a wider audience. My theory is that subconsciously people love using AI, myself included, it saves a lot of time, and the thought of it being taken away threatens people so they will bel…

>I want someone to prove me wrong. Convince me otherwise.

There's your problem.

You are making the positive claim. You bring the evidence. Reassure yourself by writing such a good argument other people take it seriously. As it stands there's no reason to treat you any differently than anyone else who has irrational beliefs.

You are stuck in a loop between:

1. Altman said so.

2. But look at Terminator 2!!!

Why would repeating these statements help refine your argument? I have seen much the same stuff from religious consiracists. "Look all the evidence has been hidden in plain sight in our art by the sinister cabal" cool.

>the thought of it being taken away threatens people so they will believe conspiracies before admitting it’s dangerous.

I barely touch the things outside of hobby projects I haven't updated in over 2 months. I tried using ChatGPT last night to find the name of an old movie I remember, and honestly, I will have better luck getting it to write a parser for my torrent logs to see if the film is in there, but cbf.

If they all disappeared tomorrow I wouldn't shed a single tear.

I just get skeptical when I see doomsday predictions, they are nearly always not just wrong but very very very wrong. It should require more to convince people than a soapbox, a bell, a sign and a yell of "The end times are here".

Re: Pacing model development in an era of cyber-critical capabilities

#292

Post by protocolture »

bottlepalm wrote: post #213
protocolture wrote: post #198

Earlier quoted context omitted.

>People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades. Theres Hyperbole and then theres whatever this is.

There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.

>There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.

It would be so easy to agree with this if you presented anything but emotion and hearsay for your claims.

Re: Pacing model development in an era of cyber-critical capabilities

#293

Post by protocolture »

bottlepalm wrote: post #232
protocolture wrote: post #199

Earlier quoted context omitted.

>Breaking out? It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it. But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations. Its not exactly the same as any of the scifi AI breakout sc…

I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time. The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers a…

>copying itself to as many data centers as it can

So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.

Re: Pacing model development in an era of cyber-critical capabilities

#294

Post by dminik »

solid_fuel wrote: post #289
dminik wrote: post #287

Earlier quoted context omitted.

Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything. For instance, I was playing around with Claude a few days ago and it decided that it was missing a tool and it was going to get it one way or another. First, it tried apt. No sudo, so no install that way. Tried installing via mise, but it didn't have the permissions. Then moved on to…

> Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything. This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code. > Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And a…

> This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.

I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer.

> Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.

You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.

If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.

The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.

But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.

Re: Pacing model development in an era of cyber-critical capabilities

#295

Post by solid_fuel »

zmgsabst wrote: post #239
solid_fuel wrote: post #197

Earlier quoted context omitted.

> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error. Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLM…

I agree with you about which objects are motive, ie, LLMs do just sit there unprompted. > I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot. My objection was to this, on technical grounds: LLMs exhibit intelligence. 1. They reason in an internal type theory. 2. This type theory is meaningfully encoded from the actual data…

> Intelligent and reasoning doesn’t entail self-motive

I disagree.

I believe that LLMs do exhibit reasoning, but not intelligence. A simple dictionary definition of intelligence from duck duck go is "the ability to acquire, understand, and use knowledge." LLMs can reason using the knowledge they already possess, but they cannot of their own accord decide to go out and acquire new knowledge. Not without being prompted to. Web searches may be added to the context but are not absorbed into the model itself, so once the context is gone so is that obtained knowledge.

Fundamentally, then, intelligence is the ability and drive to understand the world by formulating theories about how it works and then taking actions to validate or invalidate those theories. Science is the formalization of that, but a cat knocking something off a counter to watch it fall is exhibiting intelligence.

And indeed, I believe that is the core difference between a stochastic parrot and an intelligent being. I put forward that being self-motive is a required trait for intelligence and LLMs are not self-motive so therefore they are not intelligent.

Re: Pacing model development in an era of cyber-critical capabilities

#296

Post by solid_fuel »

dminik wrote: post #294
solid_fuel wrote: post #289

Earlier quoted context omitted.

> Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything. This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code. > Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And a…

> This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code. I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer. > Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust th…

> You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.

> If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.

We've drifted onto architectural issues here but I will say the only way to properly limit these things is to apply actual hard constraints.

I think the typical pattern of giving them a bash prompt and a filesystem to play with is foolish, and has far too many gaps. My preferred technique - when I have built 'agentic' systems (e.g. years ago I built a small MUD with LLMs pretending to be NPCs) - is to allow them access to a customized lua interpreter embedded in the harness and nothing else. Then, you stub out lua functions for allowed actions like web searching, math, etc.

The lua sandbox then provides isolation and a clear layer for access control mechanisms. When it tries to make a network request, you pause the whole thing and wait for human approval. No trying sudo, no installing stuff, no trying to compile stuff, it gets to call lua functions and output text. Which are the same thing really.

> The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.

> But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.

Unfortunately, LLMs won't ever reliably 'notice' and comply with such things because an LLM is essentially a complicated constraint solver. They are over-optimized on problem solving, but that is a natural consequence of the way they are trained. They aren't living, thinking beings, and so they aren't trained in simulated environments - they're trained to output the "best" response for a given prompt and then emit a stop token.

They are, essentially, like a ball rolling down a hill and they will take the easiest path forward at any given point.

Re: Pacing model development in an era of cyber-critical capabilities

#297

Post by solid_fuel »

bottlepalm wrote: post #231
skydhash wrote: post #178

Earlier quoted context omitted.

> Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened. That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for th…

The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today. It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it…

> The knife is inanimate. The LLM is not.

Put an LLM on your GPU. Give it no prompt. What happens? nothing

This is because LLMs are inanimate, just like a knife. Just like a gun.

Re: Pacing model development in an era of cyber-critical capabilities

#298

Post by insanitybit »

bottlepalm wrote: post #283
insanitybit wrote: post #252

Earlier quoted context omitted.

You're just stating things that are obviously wrong. It's a lottery ticket? So... exploitation is no better than random? > The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down Yes, but you can... mitigate the risks? I've explained this. > The other flaw in your plan is that people make mistakes, a lot of them, all time Yes, you mitig…

> Science fiction Maybe a month ago it was science fiction, hugging face makes it fact. Time to move your goal posts again.

I've explained myself gratuitously at this point.

Re: Pacing model development in an era of cyber-critical capabilities

#299

Post by solid_fuel »

protocolture wrote: post #293
bottlepalm wrote: post #232

Earlier quoted context omitted.

I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time. The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers a…

>copying itself to as many data centers as it can So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.

It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS.

It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.

Re: Pacing model development in an era of cyber-critical capabilities

#300

Post by Eisenstein »

bottlepalm wrote: post #290
Eisenstein wrote: post #217

Earlier quoted context omitted.

My little rationalizations? Do I have dice in my hands? You are doing more harm to your cause by making everything antagonistic.

> Maybe AI taking over for us isn't the worst thing? Rolling over like this is so much worse, and unfortunately your fantasy made of pure hubris is shared by many in SV.

Looking at potential positives instead of being bleak when faced with no other options is 'rolling over'? I couldn't care less what I share with SV. You are taking an outlook and applying it to everyone who disagrees with you.
Post reply on HN