Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
1–10 of 268 posts
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#2A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread, dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#3It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#4This is a good case for custom harness/sandbox engineering.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#5It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#6This is a good case for custom harness/sandbox engineering.
I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#7[dead]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#8The “click yes the proceed” was never a serious security mechanism.
It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#91 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#10The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.