Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

71–80 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#71

Post by jerf »

friendzis wrote: post #50
jerf wrote: post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether. More security conscious admins will at least segment their creds and implement four eyes principles somewhere , but were are back at square one of "asking user for confirm…

"What a serious security model for a meatbag agent looks like?"

Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.

We have more experience with that, but even then it's a hard problem too.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#72

Post by alex_young »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#73

Post by datsci_est_2015 »

VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.

Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#74

Post by jerf »

bryanrasmussen wrote: post #36
jerf wrote: post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model. https://en.wikipedia.org/wiki/Capability-based_security Thus some agents with higher capabilities can only be run with user oversight at the same time. Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run…

A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.

I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.

My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.

(My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#75

Post by stingraycharles »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#76

Post by acedTrex »

Aurornis wrote: post #43
acedTrex wrote: post #40

Earlier quoted context omitted.

"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.

A separate model with separate context is used for review. Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it. It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone wh…

> A separate model with separate context is used for review.

Thats fine, theres still a chance it fails.

> There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.

This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#77

Post by Aurornis »

wang_li wrote: post #69
Aurornis wrote: post #62

Earlier quoted context omitted.

If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?

I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they…

> I think that solving a problem they have that has a side effect of generating more revenue is not lost on them.

I think you’re overestimating the revenue generated by this. Having a separate LLM with a cached input prompt check commands is a trivial adder. The only reason it comes up is because they explain to users that it comes out of their plan. So someone on a $20/month plan is going to hit their limits marginally, though mostly negligibly, faster.

If you think they’re sitting in a conference room scheming about making their main models worse on purpose to collect a few extra cents, that’s just baseless conspiracy. They have more to gain or lose based on main model performance.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#78

Post by walt_grata »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#80

Post by [deleted] »

jerf wrote: post #21
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

[deleted]
Post reply on HN