Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

131–140 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#131
post #14

Earlier quoted context omitted.

There's really no other alternative, though?

I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?

I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#132

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

[dead]

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#134
post #21

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend.

Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by llms.

Thirdly, prevent it from sending anything but get requests to anywhere you don't want it to post stuff, with firewall configuration.

After that, turn the horrible security theater of it asking permission for anything off. So far we have had no incidents. It could of course still pull a malicious package from somewhere, that exfiltrates code using GET, but at least it can't send any credentials or user data over.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#135
post #91

Earlier quoted context omitted.

If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the L…

> This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.

Every piece of software comes with giant disclaimers like this.

You’ve just been clicking past them.

This really isn’t new.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#136

Earlier quoted context omitted.

That would be fine if Claude was open source and/or unpaid

If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?

Yes, the fine print.

Only this so-called "AI" needs it full size on the front page.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#137

Perhaps there needs to be a plugin for these tools that uses your webcam to make you Point and Call ( https://en.wikipedia.org/wiki/Pointing_and_calling ) for every single approval.

I’d give it 2 months for it to turn into a “please drink verification can to continue”-type situation.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#138
post #107
post #91

Earlier quoted context omitted.

> This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.

You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“

This sentence only exists to offset law (or case law) have gave damage rights to anyone.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#140
post #21

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

This is a complex task, and I want to avoid blatant self-promotion, but there are solutions that people are building which allows you to give a degree of freedom to your agents but also lock them down as well. Our company has a product which is just one such example. At this point it's really geared towards orgs running agents in a cluster to handle tasks, rather than e.g. making sure your claude code doesn't post your GPG keys to the blockchain or something.

In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.

You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.

We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: https://www.tigera.io/tigera-products/lynx/

The idea for Lynx is:

1. Your org has a bunch of scoped agents

2. You have a fixed list of what those agents should be doing and what they need to be accessing

3. They don't or won't need to access anything else

So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.

Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.

Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.

The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.

The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.

Post reply on HN