Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

81–90 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#81

Post by kstenerud »

Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI:

- No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox.

- Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker

- Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart (Tart lets you run simulators).

- Network control

- Secrets control (file mounts or credentials broker)

- NO ambient data (ENV is replaced with a minimal and local-to-sandbox one)

- NO access to your homedir. You have to explicitly mount things you want.

- NO direct access to your workdir: You can get a diff of the changes the agent made, and then choose whether to apply them.

- gitignored files never get copied in. The agent never sees them.

- FOSS

https://github.com/kstenerud/yoloai

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#82

Post by grumple »

applfanboysbgon wrote: post #46
csbrooks wrote: post #12

Earlier quoted context omitted.

There's really no other alternative, though?

Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.

Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#85

Post by harimau777 »

VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

In my most recent job I was definitely under that sort of timer.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#86

Post by wredcoll »

jerf wrote: post #21
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts!

As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#87

Post by zdragnar »

jerf wrote: post #74
bryanrasmussen wrote: post #36

Earlier quoted context omitted.

I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model. https://en.wikipedia.org/wiki/Capability-based_security Thus some agents with higher capabilities can only be run with user oversight at the same time. Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run…

A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities. I've done some stabby s…

What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.

Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#88

Post by cube00 »

It would have been nice if the game had disclosed that player's actions were being collected for future research.

You don't get any notice or choice it just beams it all up silently in a POST request at the end:

    "timeline": "ex01:N,ob06:Y,s14:N,sc10:N,s02:Y,s04:N,ex09:N,s10:Y,sc15:N"

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#89

Post by Aurornis »

I suggest everyone look at the game to put this in context, because it's most likely not what you think it is. https://llmgame.scalex.dev/

This is how it opens:

> 1 MINUTE UNTIL YOUR NEXT MEETING

> Claude Code is finishing up your refactor.

> It needs your approval for a few commands. Can you finish in time?

> Your eyes are already glazing over. Can you stay sharp?

It says the goal is "as many as you can"

I won the first time I played by answering 0 questions and doing nothing at all. The title screen tells you to answer as many as you can, but answering nothing at all is the easiest way to win.

If you start answering questions, thinks like 'npm run build' will get marked as dangerous. If you would have run that in your own console, you are a dangerous developer I guess. Ironically in an LLM harness it would have been sandboxed at least.

It's inconsistent, though. Other 'npm run' commands are not marked as dangerous, which is not a safe assumption if you're familiar with how npm works.

In my clicking through of the game and playing it, I had 2 runs where I succeeded (by doing nothing or little at all) and 1 run where I lost because I clicked yes to see what would be counted. Close to that 1/3 number they cited, and I guess I'm included in those stats now.

This project feels like bait dressed up as a study.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#90

Post by solenoid0937 »

gmadsen wrote: post #39
jerf wrote: post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well

Surprised no one is talking about auto mode, it solves this problem.
Post reply on HN