It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
11–20 of 268 posts
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#12It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#13It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#14Earlier quoted context omitted.
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
There's really no other alternative, though?
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#15This is a good case for custom harness/sandbox engineering.
I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#16The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#17The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#18Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#19Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#20Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.