Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

141–150 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#141
post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend. Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by…

The way our Claude Codes are configured at work is pretty nice. There are directory patterns it can't access, like .local or .config, so when it needs to it creates a throwaway scratch pad and asks you to put stuff there; screenshots, text files, command output, etc.

I was having it diagnose a GNOME extension and had to get it a copy of the code to work on; it would then write out a Python script to do the patching (which I could inspect beforehand) and have me execute it.

Not having access to .local or .config can be irritating sometimes, but it's nice to know it's not just going to exfiltrate my docker or gcloud credentials.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#142
post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend. Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by…

After watching a mid-tier offering chain together tools like it was a gorilla escaping the zoo I just gave the model its own box. I don’t have time to deal with that kind of nonsense.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#143
This mechanism is going to be the breaking point for Claude and Codex.

The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.

So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.

In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#144

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

Yep, file it in the same folder as "Terms and Conditions" notices.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#145

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all.

What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases.

For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#146

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

[flagged]

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#147
post #74

Earlier quoted context omitted.

I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model. https://en.wikipedia.org/wiki/Capability-based_security Thus some agents with higher capabilities can only be run with user oversight at the same time. Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run…

A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities. I've done some stabby s…

> My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.

You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.

You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].

The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:

brand.sig:

    signature BRAND =
    sig
        type token
    end

mint.sig:

    signature MINT =
    sig
        include BRAND
        val mint : unit -> token
    end

makebrand.fun:

    functor MakeBrand () =
    struct
        type token = unit ref
        fun mint () = ref ()
    end

filesystem.sig:

    signature FILESYSTEM =
    sig
        type token
        val readFile  : token -> string -> string
        val writeFile : token -> string -> string -> unit
    end

filesystem.fun:

    functor FileSystem (B : BRAND) :> FILESYSTEM where type token = B.token =
    struct
        type token = B.token

        fun readFile (_ : token) (path : string) : string =
            "contents of " ^ path

        fun writeFile (_ : token) (path : string) (_ : string) : unit =
            ()
    end

trusted_fs_setup.sml:

    local
        structure FileAuthority :> MINT = MakeBrand ()
    in
        structure FS :> FILESYSTEM = FileSystem (FileAuthority)
        val rootFileToken : FS.token = FileAuthority.mint ()
    end

trusted_fs.cm:

    Library
        signature FILESYSTEM
        structure FS
        val rootFileToken
    is
        brand.sig
        mint.sig
        makebrand.fun
        filesystem.sig
        filesystem.fun
        trusted_fs_setup.sml

Now for any given library using the trusted_filsystem library:

  val doc = FS.readFile rootFileToken "/etc/motd" (* Works fine *)
 
Delegation is function application:

  fun helper (t : FS.token) = FS.readFile t "log.txt"
  val log = helper rootFileToken  
And these all fail:

  val fake : FS.token = ref () (* Trying to forge a token *)
  val t = FileAuthority.mint () (* Trying to bypass the trusted kernel in trusted_fs_setup.sml by calling the mint *)
  
  (* Trying to self-issue authority by making our own brand and mint *)
  structure MyCap = MakeBrand ()
  val t : FS.token = MyCap.mint ()   (* type mismatch *)

What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.

You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.

[1] - https://en.wikipedia.org/wiki/E_(programming_language)

[2] - https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#148
post #115

Earlier quoted context omitted.

Been working on something like that for years: https://www.firefly-lang.org/

if firefly has no nulls, how do you indicate that a value is unset?

In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null.

In Rust an Option is a separate thing that you need to disambiguate to use. For example:

    match result {
        // The division was valid
        Some(x) => println!("Result: {x}"),
        // The division was invalid
        None    => println!("Cannot divide by 0"),
    }
Likewise in Rust, you can't have a null pointer, but you can have an Optional pointer, which is either a pointer to something or is not anything.

Firefly seems to have a similar case structure, though the first example I could find is in the Exceptions section: https://www.firefly-lang.org/reference/exceptions

    grabOption[T](option: Option[T]): T {
        | Some(v) => v
        | None => throw(GrabException())
    }

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#149
post #94

Earlier quoted context omitted.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

I guess I'm confused; why would you, as an admin, want to allow your users to use less secure passwords? I get that your argument is that all traffic is local and so complex passwords are unnecessary (debatable, I would hold that a strong password policy would still be desirable for defense-in-depth, depending on the network and risk profile), but that doesn't make them undesirable.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#150

Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI: - No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox. - Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker - Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart…

Ah yes sandbox it because Docker has never experienced a CVE.

Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.

Post reply on HN