Anyone else play Warhammer 40k? I went into this article really excited for a genius war game bot haha.
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
51–60 of 268 posts
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#52Earlier quoted context omitted.
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it. The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things. The common harnesses also support remot…
"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.
"Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#53Sandbox and use Local AI. This is the real answer.
Yet the AI can still escape the "sandbox", unless it is physically unable to connect to another computer and completely airgapped.
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#54The future of software is fixing bugs and security issues in production.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#55So humans scored 66% on human eval?
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#56I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…
Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#57It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#58A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then. It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored. I also incorporated the feedback and insig…
In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?
I think there are too many problems with HITL that even a simple experiment like this game shows. The fatigue causes people to jump to complete bypasses instead, and we need to work more on raising the general awareness of the new types of threats (which is also evolving rapidly). We can't point to it as a valid solution. A way could be to make sandboxing and context/permission isolation easier from the tooling and only give these wide ranged accesses once these are in place than to consider HITL an acceptable alternative
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#59I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…
Also a lot of them may or may not be safe depending on additional context that you don't have in the test.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#60It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.