Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

261–268 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#261

Post by ratelimitsteve »

pllbnk wrote: post #44
VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandbox…

this. the problem with HitL is that there are so many decisions to make that decision fatigue is inevitable. Beyond a certain number of approvals most people are going to switch to an approval process that goes from "carefully consider the need and risk of each command" to "approve unless there's a blatant sudo rm -rf / because the last several hundred times i approved without reading it was fine"

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#262

Post by zdragnar »

bryanrasmussen wrote: post #202
zdragnar wrote: post #198

Earlier quoted context omitted.

I've read this a few times, and it still sounds like you're trying to re-invent an operating system. You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. I can imagine all sorts of ways to "color" functions by what capabilities they use, and thus statically analyzing the…

>You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. well, you can could of course go ahead and read some of the literature on capabilities based security, but you don't seem interested in that, so I guess I will give a go. Here I will give a not necessarily great descri…

I read it in your earlier post, but it didn't really address OP's arguments. It's basically an enforced mediator pattern. Even the article summary points out the language design doesn't provide sufficient security to avoid DOS on the same machine, only for avoiding flooding the network with messages between machines.

As for your solitare example, it goes back to my argument that you're stuck designing both the language and an OS for it, because the problem exists not in the design of the language (grammar, syntax, etc) but in how the runtime interfaces with the host operating system.

No two operating systems have the same semantics or capabilities, and, as I pointed out, even the notion of what defines a file, filesystem or a user is not consistent. You're not going to be embedding those things into the language, but the runtime.

At the end of the day, it's just a question of how do I add color to my code for an analyzer to pick up on it. Decorating functions (and operators) isn't novel, nor is analyzing code. All the novel, hard work is going to be done in the runtime or compiler.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#263

Post by zbentley »

godelski wrote: post #259
akdev1l wrote: post #248

Earlier quoted context omitted.

from subprocess import run from sys import argv source = argv[1] destination = argv[2] run([ "rsync", "-zavp", source, destination ]) vs #!/usr/bin/env bash source="$1" destination="$2" rsync \ -zavp \ "$source" \ "$destination" the overhead is not really that bad imo

That's not really what people mean. Your example isn't accurate because you just ran bash two different ways.

Huh? The Python example does not use Bash or the shell in any way.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#264

Post by zbentley »

tremon wrote: post #252
doublerabbit wrote: post #246

Earlier quoted context omitted.

> We have vastly better tools now in those niches. But do we? I'll beg to differ. Python explodes in to LoC when you try to attempt to work with the OS. You need to source modules that turn the application in to bloat. 98% of def's you don't require. If you need a multi-purpose tool, or a model than sure. For the simple automation of running a rsync or ZFS snapshot schedule to copy to & from a NAS in a guaranteed tim…

For anything more complex than linear shell scripting, I prefer ruby --disable-gems these days. As you say, Python is horrible for composing and executing shell commands, but Ruby's FileUtils#sh (combined with %W{} array literals, %x{} output capturing and regular #{} string interpolation) provides a pretty low-impedance interface.

FileUtils#sh doesn't exist:

    ruby --disable-gems -e 'require "FileUtils"; FileUtils::sh("ls")'

    -e:1:in `': undefined method `sh' for FileUtils:Module (NoMethodError)

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#266

Post by akdev1l »

godelski wrote: post #259
akdev1l wrote: post #248

Earlier quoted context omitted.

from subprocess import run from sys import argv source = argv[1] destination = argv[2] run([ "rsync", "-zavp", source, destination ]) vs #!/usr/bin/env bash source="$1" destination="$2" rsync \ -zavp \ "$source" \ "$destination" the overhead is not really that bad imo

That's not really what people mean. Your example isn't accurate because you just ran bash two different ways.

subprocess.run() uses Popen and defaults shell=False. There is no shell process involved. As the other user already pointed out.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#267

Post by drob518 »

noinsight wrote: post #226
drob518 wrote: post #61

This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things…

Like the fabulous Windows UAC dialog. Perhaps the worst dialog in history.

Yep, exactly. Engineers always think these sorts of “ask the user what to do” mitigations will be effective, but they forget that users have no clue and the repetition causes users to tune them out. And I say that as an engineer.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#268

Post by StilesCrisis »

godelski wrote: post #224
StilesCrisis wrote: post #211

Earlier quoted context omitted.

My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing. Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.

My favorite is Claude finding .git/index.lock, asking to remove it, finding out it no longer exists, and then hitting a lock again that it itself created. Poor little robot, stop shooting yourself in the foot.

Could be worse! Claude just assumes repos all use git, and will try to run git commands against a Perforce repo unless your CLAUDE.md gives it strict instructions. When the git commands fail, it will just use chmod to make the files writable. (P4 marks files as not-writable until you check them out.)
Post reply on HN