Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

191–200 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#191

Post by Terr_ »

dgunay wrote: post #172

For me the problems with agent permission prompts are twofold: 1) I generally have a lot of things where I am okay with the agent calling a specific tool (maybe in certain ways) as much as it wants. This allowlisting approach is often defeated by the model's own proclivity to get fancy with inline scripting. 2) Checking for intent/alignment of the agent is the primary reason I still even use permission prompts, becau…

> This allowlisting approach is often defeated

See also: https://gtfobins.org/

> GTFOBins is a curated list of Unix-like executables that can be used to bypass local security restrictions in misconfigured systems.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#192

Post by Terr_ »

not-kinsale-joe wrote: post #32

I think there is potential for a good video game, Papers Please style, where you are a human in the loop.

Between US federal immigration "enforcement" and workplace "AI workflows", I think that style of dystopic game becomes uncomfortably close to real life twice-over...

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#193

Post by user43928 »

solenoid0937 wrote: post #90
gmadsen wrote: post #39

Earlier quoted context omitted.

Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well

Surprised no one is talking about auto mode, it solves this problem.

I never had an issue with Auto mode in Claude Code.

It used to be the same with Codex, until one day it became entirely unusable, rejecting even git operations out of concern for the privacy settings of my repository that it "cannot verify".

Maybe worth to note that this is the only way the feature gets in the way. If on the contrary they accidentally make Auto behave like Full Access, we would never notice.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#194

Post by mafuy »

tcdent wrote: post #150
kstenerud wrote: post #81

Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI: - No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox. - Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker - Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart…

Ah yes sandbox it because Docker has never experienced a CVE. Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.

It's better than the alternative. Don't complain about someone offering an imperfect improvment, if you don't have something even better to offer.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#197

Post by walrus01 »

It would be an interesting comparison to compare the human "miss rate" shown in the table there with the exact same tests repeated with a different LLM watching and approving or denying each action. No human in the loop, just record the results and take the measurement of pass/fail at the end of the run. With something fairly large and smart that has been given a very specific system prompt to watch and prevent harmful actions or data leaks.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#198

Post by zdragnar »

jerf wrote: post #111
zdragnar wrote: post #87

Earlier quoted context omitted.

What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags. Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the langua…

In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is…

I've read this a few times, and it still sounds like you're trying to re-invent an operating system. You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.

I can imagine all sorts of ways to "color" functions by what capabilities they use, and thus statically analyzing the program to assert what it does, but again, all of the core capabilities would be provided by the standard library / builtin functions, so it seems to not add much to the actual design of the language beyond ensuring all control flow, redirection and reflection can be statically analyzed.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#200

Post by electric_toucan »

kstenerud wrote: post #81

Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI: - No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox. - Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker - Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart…

Sandboxing is useful but usually not a replacement for permission prompts. If you give it network access, it could still run destructive commands against allowed domains, for example
Post reply on HN