Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

221–230 of 269 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#221

This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.

Getting behind the wheel of an F1 car on a track involves lots of proving time that you can actually handle such a vehicle. Meanwhile anyone with a credit card can grant an AI system to impersonate their access as a starting point.

There are places/services where you can drive a (few years old, and slightly detuned, but not THAT much) F1 car with little more than a (rather high limit) credit card.

https://www.lrs-formula.com/en/ for instance.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#222
post #218

Earlier quoted context omitted.

No, it was testing in the context of a kind of coding I simply don't do. If an AI harness asked me to permit just one of maybe half of the suggested commands, I would stop the harness since something has gone very wrong. To be clear: I work in C and Python. It's asking to run npm. That's immediately the end of that run and the start of the search for a better setup. (My work does not overlap with anything in npm/JS/w…

So then, you are a bottleneck. You will only review things that fit within your preferred small niche and area of responsibility. You cannot oversee increasing amounts of automation covering larger areas, because that would mean you are no longer 'working in C and Python' as you have to deal with things that are not '2 layers below HTTP', and you will not deal with anything that might involve, say, web dev, despite t…

Spoken like a true journalist with little dev experience :). Though I don't understand why you felt it necessary to go borderline insulting/condescending? Ohwell.

Funnily enough, yeah, I'm a bottleneck… though for the exact opposite of the reason you're giving. My employer doesn't do job titles, elsewhere I'd be a distinguished or principal engineer. I have about 22 years of depth of knowledge in my area. I'm there to answer questions, guide design decisions, review things and provide context. There isn't enough of me to handle all that, so, yeah, bottleneck. Which is solved by people around me getting things out of the way when they aren't related to my "scope" as you say, so I don't waste time on them.

But even for a junior dev, your comment makes little sense. You need to learn breadth and depth. If you're working at a project, you should rather stick to the existing tooling and go into depth, especially if a bunch of senior devs have put time and thought into finding a local optimium.

So, then, does your comment make sense for a midrange dev? Maybe. But even then, only if they're not having to get shit done. Your AI agent doesn't get to decide to introduce you to new tooling or a new ecosystem midway through a sprint. You do that shit when you're in the initial exploratory phase for a project, or tinkering with a pet project.

Maybe journalism is very focused on breadth? I don't know. I thought journalists also have their in-deep areas of knowledge? AFAIK your area of knowledge is cryptocurrencies and related — if the AI agent asked you a safety question about fiat banking, what would you do?

P.S.: 2 layers below HTTP is IP. I hope I didn't introduce a misunderstanding there, I'm not in a project where 2 layers above me there is HTTP. There is nothing above. That's just where we are, IP. We don't work on anything that is HTTP, anywhere in the entire company. (Well, sure, we have a website, but if I were to even think about spending time on that somebody would blow a gasket.)

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#223
post #44

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandbox…

  > I think minority do
Let's be real, the minority of people understand bash. It is a terrible language with tons of footguns[0], albeit a very useful language and still worth learning. The number of ~/.claude/settings.json I've seen with "permissions": { "allow": ["Bash(find *)"]} I've seen is crazy[1]. Hell, most people I talk to think `find` is a tool that is used for searching for files. I mean... it does that... along with arbitrary code execution ¯\_(ツ)_/¯

[0] https://mywiki.wooledge.org/BashPitfalls

Edit:

[1] Fuck it, here's the lazy search: https://github.com/search?q=%22Bash(find%20*)%22&type=code

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#224

Earlier quoted context omitted.

If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wis…

My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing. Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.

My favorite is Claude finding .git/index.lock, asking to remove it, finding out it no longer exists, and then hitting a lock again that it itself created. Poor little robot, stop shooting yourself in the foot.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#225
post #59

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

Also a lot of them may or may not be safe depending on additional context that you don't have in the test.

Isn't that kinda the point though?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#226
post #61

This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things…

Like the fabulous Windows UAC dialog. Perhaps the worst dialog in history.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#227

Earlier quoted context omitted.

That would be fine if Claude was open source and/or unpaid

If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?

An outage is not the same thing. A similar thing would be if Outlook 365 started sending emails on your behalf or changing around words in them, or if SAP / Quickbooks said: "No guarantee that the numbers you type in match the results of the annual statement we produce", or if Tesla was like "You should absolutely not use our cars for driving anywhere".

If you bring language like this into more traditional fields, it feels absolutely bonkers.

edit: some lawsuits:

* Air Canada forced to honour chatbot offer - https://lsj.com.au/articles/air-canada-forced-to-honour-chat...

* Tesla pays $329M for defective autopilot - https://www.anzalonelaw.com/jury-finds-tesla-autopilot-defec...

* but then again: non-performance & damages disclaimer upheld in the "Diedrich" lawsuit https://www.lexology.com/library/detail.aspx?g=8f61a973-f05c...

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#229
Because things like this, I am using sandboxes, doing automated and manual review before running any code. As it takes time, I now need to try to one shot the development of the code which takes longer to do a proper specification.

And even with much care on all the steps, when I read carefully, I still find wrong things at multiple levels.

When folks like uncle Bob says they don't review AI code, I can only think they are burnt out or being unprofessional.

Indeed it saves some time writing the code, but overall, I think we just moved concerns from one place to another. When writing your code, you are automatically reviewing things and integrating with other pieces. Ofc, sometimes mistakes happen, but my impression is that important code still takes about the same to develop.

There are ways to go further and still try to LLM'it all the way, but since claude 4.8 the types of mistakes have been much more convoluted. Fable and opus 5 leaves too many gaps and take so many poor decisions.

GLM has a nice balance. It stops me only when important things come up and it integrates well with my way of working.

With Anthropic is a lot of do a lot, clean up and fix fix fix. With glm has been more like working together and delivering...

Anthropic models have gone wrong.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#230
post #6
post #4

This is a good case for custom harness/sandbox engineering.

I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".

True. Real harness engineering is when you actually measure outcome statistically. Although i would say 5 samples might be enough to conclude something
Post reply on HN