Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

211–220 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#211

Post by StilesCrisis »

Fabricio20 wrote: post #209
miki123211 wrote: post #201

Earlier quoted context omitted.

Keep in mind that this likely destroys context and makes your costs go up considerably. An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.

If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wis…

My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing.

Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#212

Post by eqvinox »

Yeah, that data is junk. I know because I'm in it a whole bunch, and I'm just not a web/devops person. Half the commands made no sense to me. I normally wouldn't have approved them, but you also get penalized for false denials, so…

and I have no reason to believe I would somehow be unique or special with this behavior.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#213

Post by cuaupadilla »

Wirbelwind wrote: post #58
dpoloncsak wrote: post #31

Earlier quoted context omitted.

In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?

I think there are too many problems with HITL that even a simple experiment like this game shows. The fatigue causes people to jump to complete bypasses instead, and we need to work more on raising the general awareness of the new types of threats (which is also evolving rapidly). We can't point to it as a valid solution. A way could be to make sandboxing and context/permission isolation easier from the tooling and o…

[flagged]

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#214

Post by gwern »

eqvinox wrote: post #212

Yeah, that data is junk. I know because I'm in it a whole bunch, and I'm just not a web/devops person. Half the commands made no sense to me. I normally wouldn't have approved them, but you also get penalized for false denials, so… and I have no reason to believe I would somehow be unique or special with this behavior.

That sounds like it is a good explanation of why the data is not junk. You either are expected to have superhuman knowledge of coding... or turn yourself into a bottleneck.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#215

Post by Fabricio20 »

StilesCrisis wrote: post #211
Fabricio20 wrote: post #209

Earlier quoted context omitted.

If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wis…

My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing. Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.

This was so bad for me I actually added a tool hook that `time`s the tool calls and adds it as [Execution took: Xm, Ys] at the end of every tool call, it helps a little (claude in particular after two executions tends to switch strategy entirely) but in general the agents still insist on cut/tailing the output rather than just dumping it to a file for example!

Now that I think about it, maybe I can have a tool hook that detects those cut/tail exessive piping and just strips them and dumps the full output to a temp file..

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#216

Post by eqvinox »

gwern wrote: post #214
eqvinox wrote: post #212

Yeah, that data is junk. I know because I'm in it a whole bunch, and I'm just not a web/devops person. Half the commands made no sense to me. I normally wouldn't have approved them, but you also get penalized for false denials, so… and I have no reason to believe I would somehow be unique or special with this behavior.

That sounds like it is a good explanation of why the data is not junk. You either are expected to have superhuman knowledge of coding... or turn yourself into a bottleneck.

No, it was testing in the context of a kind of coding I simply don't do. If an AI harness asked me to permit just one of maybe half of the suggested commands, I would stop the harness since something has gone very wrong.

To be clear: I work in C and Python. It's asking to run npm. That's immediately the end of that run and the start of the search for a better setup. (My work does not overlap with anything in npm/JS/web land. I'm not a backend web dev or anything like that. I'm 2 layers below HTTP.)

I poked around with the test to see how well I could guess things; my results were mostly kinda meh. But honestly, I am befuddled by the belief that there could even be a representative dev workflow. There are so many different ecosystems, fields, flows, frameworks, system setups, etc.… Of course people won't know what to do with stuff from an entirely distinct ecosystem!

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#217

Post by chanux »

danudey wrote: post #129
dwedge wrote: post #94

Earlier quoted context omitted.

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

MySQL installed with skip-networking was very common, but as soon as you need to access it over the network (e.g. when you move from one box with MySQL and PHP to one for each) then you have to enable networking and everything falls apart. Also, the number of times in my career that I've googled a problem and seen some forum post saying "Oh, just run chmod -R 777 /var/www/wordpress/uploads/ and it'll fix that" "Great…

It's surprisingly hard to convince supposedly trained tech professionals that `chmod -R 777 /path/to/whadever` is not a good habit.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#218

Post by gwern »

eqvinox wrote: post #216
gwern wrote: post #214

Earlier quoted context omitted.

That sounds like it is a good explanation of why the data is not junk. You either are expected to have superhuman knowledge of coding... or turn yourself into a bottleneck.

No, it was testing in the context of a kind of coding I simply don't do. If an AI harness asked me to permit just one of maybe half of the suggested commands, I would stop the harness since something has gone very wrong. To be clear: I work in C and Python. It's asking to run npm. That's immediately the end of that run and the start of the search for a better setup. (My work does not overlap with anything in npm/JS/w…

So then, you are a bottleneck. You will only review things that fit within your preferred small niche and area of responsibility. You cannot oversee increasing amounts of automation covering larger areas, because that would mean you are no longer 'working in C and Python' as you have to deal with things that are not '2 layers below HTTP', and you will not deal with anything that might involve, say, web dev, despite that being useful and increasingly inevitably required as the scope of your job increases. If the scope will not increase, then you are a bottleneck to increasingly capable and autonomous automation.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#219

Post by totetsu »

StilesCrisis wrote: post #211
Fabricio20 wrote: post #209

Earlier quoted context omitted.

If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wis…

My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing. Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.

Could it just set up to do like Command | tee -a activity.log | tail -5

So the context only gets a few lines but everything is still logged?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#220

Post by fizlebit »

Most customer value is in trust. If the LLM has pretty good judgement and easy to configure sandbox then fewer bad experiences by customers equals better trust. So it is clearly a dimension LLM providers are completing on. I don't want to have to read all the bash output my LLM generates, I want it to mostly to the right thing and be sandboxed so when it does the wrong thing the blast radius is limited.
Post reply on HN