It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.
[flagged]
LLM wrote this comment.
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
171–180 of 268 posts
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#172For me the problems with agent permission prompts are twofold:
1) I generally have a lot of things where I am okay with the agent calling a specific tool (maybe in certain ways) as much as it wants. This allowlisting approach is often defeated by the model's own proclivity to get fancy with inline scripting.
2) Checking for intent/alignment of the agent is the primary reason I still even use permission prompts, because IME it's way more common for the agent to destroy information that you didn't want it to destroy than for it to be tricked into exfiltrating secrets. However it's very easy to fatigue out of it because having even the smallest bit of tool call restrictions means that #1 leads to never ending permission prompts. Claude Code's "auto mode" doesn't help here because AFAIK it is looking for security threats, not the model misinterpreting my intent, and it can't be tuned to look for things like "please gate tool calls which may delete data."
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#173[flagged]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#174I remember when that game was posted and I do believe such result
my personal experience was that I do not have "I don't know what that is, so not allowed" as a default...
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#175[flagged]
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#176Earlier quoted context omitted.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…
Maybe that is why so many things live in Excel instead of a proper database. Security at the expense of convenience does come at the expense of security, after all. Not sure why passwords still exist conceptually. I was hoping we'd move past this annoyance, but instead security has become even more annoying. And all that security with two factor hoops to jump through only for someone to steal your session cookie.
What would passwords move past too? A device people lose? Some specifications everyone argues about and implements differently. Just look at SAML and everyone using slightly different terminology.
Security is more annoying because the attacks are better than ever.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#177Earlier quoted context omitted.
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html
Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#178Earlier quoted context omitted.
Yep and the car wasn't self-driving at the moment it crashed.
...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash.
https://www.nhtsa.gov/laws-regulations/standing-general-orde...
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#179I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…
The point is more that it's non-trivial and should not be treated as easily accomplished.
Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
#180Earlier quoted context omitted.
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
Well it turns out that we have yet another sandbox escape just released today called "Zapscape". My point is if an agent recited how to find one in its memory or training set and it is air-gapped, the chances of it spreading and infecting other computers is pretty low. [0] https://news.ycombinator.com/item?id=49198843
Gonna have to point out that's a KVM CVE. I was very specific about using a bytecode interpreter.
If you're serious about a secure sandbox, you don't touch hardware virtualization with a 10 foot pole. In fact, you don't even use an emulator that lowers code into native machine code like QEMU. The standard for secure sandboxes is Bochs: https://github.com/bochs-emu/Bochs
Not that Bochs is perfect, a new CVE was discovered back in June. But that's the 5th CVE it's had in its lifetime, and it has a much smaller upper bound on possible CVEs compared to something like KVM or QEMU.
The reason why you use something like this isn't just for the security you get out of it, but also the deep introspection and analysis facilities you get out of it as well. Unless you're a very well funded lab, it's actually quite hard to do analysis on bare metal when you can't trust your own kernel. You can always airgap the host machine (and good defense in depth does), but that's still not an appropriate sandbox by itself, even if it's theoretically secure.