Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

41–50 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#41

Post by lelandfe »

VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

The most fundamental flaw in the test is that we know we're taking a test.

How many devs take this adversarial a stance to their work?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#42

Post by jstummbillig »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#43

Post by Aurornis »

acedTrex wrote: post #40
Aurornis wrote: post #35

Earlier quoted context omitted.

Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it. The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things. The common harnesses also support remot…

"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.

A separate model with separate context is used for review.

Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.

It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#44

Post by pllbnk »

VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!

We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.

If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#45

Post by cedilla »

crabbone wrote: post #25
jascha_eng wrote: post #9

1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.

1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this…

1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#46

Post by applfanboysbgon »

csbrooks wrote: post #12
applfanboysbgon wrote: post #11

Earlier quoted context omitted.

The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.

There's really no other alternative, though?

Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#47

Post by Kinrany »

lelandfe wrote: post #41
VladVladikoff wrote: post #19

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

The most fundamental flaw in the test is that we know we're taking a test. How many devs take this adversarial a stance to their work?

It doesn't matter if the results are bad even when the devs know that it's a test.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#48

Post by prpl »

hobofan wrote: post #38
prpl wrote: post #27

Earlier quoted context omitted.

For one, I’ve been working on a generic sandbox environment github.com/brianv0/formwork You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.

> files, network, and MCP tools Locking that down to nothing is trivial for any harness: just don't expose those to the LLM. The tricky part is allowing access to those.

sure it’s not tricky. But everybody does it different and OpenAI couldn’t even be bothered to do it right when benchmarking their models

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#49

Post by kaicianflone »

What is the professional consensus on AI governance?

It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.

Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.

Or are these risks overstated?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#50

Post by friendzis »

jerf wrote: post #21
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.

More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".

Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.

Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.

While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.

Post reply on HN