Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

201–210 of 269 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#201
post #44

Earlier quoted context omitted.

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandbox…

For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.

Keep in mind that this likely destroys context and makes your costs go up considerably.

An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#202
post #111

Earlier quoted context omitted.

In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is…

I've read this a few times, and it still sounds like you're trying to re-invent an operating system. You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. I can imagine all sorts of ways to "color" functions by what capabilities they use, and thus statically analyzing the…

>You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.

well, you can could of course go ahead and read some of the literature on capabilities based security, but you don't seem interested in that, so I guess I will give a go.

Here I will give a not necessarily great description of capability based security where the capabilities are handled on the OS level, this is my poorly remembered version of a document read about 20 years ago.

In Windows if you, as an attacker, are somehow able to do a complicated attack on any program that a user runs and it succeeds you can make that program do anything that the user has access to do. For example make it write to the local filesystem. For a large number of programs writing to the local filesystem does not make any sense and they were not written to do that, but now that the program has been corrupted by your attack it will write to the filesystem.

But let us suppose our program, Solitaire, was written with a capabilities based language then in the manifest for the program when we installed it it would not give access to the file system. The capabilities system can in fact be very explicit, and you could easily have a program be explicitly limited to only those abilities that are needed to run your solitaire game, but instead of going on at length on this and the abilities it confers to do interesting things with programs because you have actually enhanced security I will just focus on the filesystem.

When the attacker attacks Solitaire that the user is running and corrupts it, it no longer has just the security model of old Windows to worry about (this program can do anything the user can, like write to the user's contacts list, send emails as the user, format the hard drive, whatever) but also the security model, enforced at the OS level, of what kind of thing the program can do. Since Solitaire was written with the limited number of capabilities needed for doing a solitaire type game it can't do any of those things the user can unless, as well as attacking the solitaire game and gaining control of that, you are also able to attack the OS level capabilities administrator and get it to give your pwned executable - Solitaire - the capabilities you explicitly want it to have.

This is one example of how capabilities based security differs from Access based capabilities where if your solitaire game is allowed to if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. (here I have switched from solitaire in Windows to a similar game in Linux but I assume you can see the same princiles hold true)

I earlier mentioned this document Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html, it shows some other examples of how a capability based security model can be used to defend against trust based attacks in shared resource systems etc.

This is a more computer science based description than the one I gave and more abstract than my example of someone attacking Solitaire in order to write to a file somewhere, but in being more abstract also clearly applicable to many other forms of threats than my very specific (and thus unlikely) example is. Also it is by Douglas Crockford and quite a bit more entertaining than what I wrote.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#203

Earlier quoted context omitted.

For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.

Keep in mind that this likely destroys context and makes your costs go up considerably. An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.

I guess ideally you'd want to force it to pipe each step back into the agent/IDE for display (and optionally wait for approval) rather than shoving it into the context. And with some sort of heuristic to skip commands that are piped into sed and similar.

or use just use actual files instead of pipes with a permissions dialogue gate on write. That would work already in Claude I think, and with IDE integration it would show each step as it went.

Of course buffering every step of a pipeline will have its own side-effects.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#204
post #100

Earlier quoted context omitted.

I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts! As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.

On unix your email account is part of the filesystem.

Yes, you're right, it would have been more specific to talk about the user/root distinction that people claimed would save you from all sorts of things.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#205

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

It’s also a game and I’m not a scientific survey…

If anything, maybe the disagreement about what is or isn’t safe says a lot to begin with.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#206

A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then. It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored. I also incorporated the feedback and insig…

How good at the game is the auto-mode classifier?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#209

Earlier quoted context omitted.

For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.

Keep in mind that this likely destroys context and makes your costs go up considerably. An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.

If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wish.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#210
post #71

Earlier quoted context omitted.

What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether. More security conscious admins will at least segment their creds and implement four eyes principles somewhere , but were are back at square one of "asking user for confirm…

"What a serious security model for a meatbag agent looks like?" Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human. We have more experience with that, but even then it's a hard problem too.

It's very similar to trying to secure a human who has infinite tolerance for risk, zero empathy, and no sense of self-preservation.

In other words, a toddler who was given a sword.

Post reply on HN