Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

31–40 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#31

Post by dpoloncsak »

Wirbelwind wrote: post #2

A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then. It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored. I also incorporated the feedback and insig…

In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#33

Post by dannyw »

kibwen wrote: post #6
whazor wrote: post #4

This is a good case for custom harness/sandbox engineering.

I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".

The word and use of "engineering" predated what you describe as "real engineering work".

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#35

Post by Aurornis »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.

The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.

The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.

If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#36

Post by bryanrasmussen »

jerf wrote: post #21
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model.

https://en.wikipedia.org/wiki/Capability-based_security

Thus some agents with higher capabilities can only be run with user oversight at the same time.

Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.

Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."

Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.

Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#37

Post by Razengan »

This brings me back to something I have always thought was lacking in OS security permissions architectures:

WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???

We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"

macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#38

Post by hobofan »

prpl wrote: post #27
jerf wrote: post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

For one, I’ve been working on a generic sandbox environment github.com/brianv0/formwork You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.

> files, network, and MCP tools

Locking that down to nothing is trivial for any harness: just don't expose those to the LLM.

The tricky part is allowing access to those.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#39

Post by gmadsen »

jerf wrote: post #21
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#40

Post by acedTrex »

Aurornis wrote: post #35
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it. The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things. The common harnesses also support remot…

"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self"

This is such an unserious approach.

Post reply on HN