This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
61–70 of 268 posts
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#62Earlier quoted context omitted.
"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.
"Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#63I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#64[flagged]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#65It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…
[dead]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#66Earlier quoted context omitted.
What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…
I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model. https://en.wikipedia.org/wiki/Capability-based_security Thus some agents with higher capabilities can only be run with user oversight at the same time. Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run…
[dead]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#67It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
That is exactly the reason why we have lonters, errors and limitations when programming.
Bc the "give a check by hand" or the "unbound discipline" never works.
The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#68I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…
That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised.
I try to cover both of the previous points in the article and they were some of the most interesting learnings for me
As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#69Earlier quoted context omitted.
"Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#70love this so much!