Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

161–170 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#161

Post by OroPla »

brunoborges wrote: post #57
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

Maybe that is why so many things live in Excel instead of a proper database. Security at the expense of convenience does come at the expense of security, after all.

Not sure why passwords still exist conceptually. I was hoping we'd move past this annoyance, but instead security has become even more annoying. And all that security with two factor hoops to jump through only for someone to steal your session cookie.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#162

Post by dwedge »

tredre3 wrote: post #152
dwedge wrote: post #94

Earlier quoted context omitted.

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

> selfhosted gitlab - there was no option to reduce password complexity for my users Gitlab's default requirements aren't that intense, but you can make them more stringent if you want. https://docs.gitlab.com/user/profile/user_passwords/#passwor... I guess your goal was to allow users to have 4 character passwords, i.e. "love." which afaik you can't do.

They must have changed that or I was completely off base because I was just trying to remove special characters and drop it from 12 (I think) and I saw them close a feature request saying they didn't think they should support it. It's possible I misunderstood and the feature request was as you say

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#163

Post by hinkley »

I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.

The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.

It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.

So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#164

Post by stonedivot »

This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#165

Post by dwedge »

brendoelfrendo wrote: post #149
dwedge wrote: post #94

Earlier quoted context omitted.

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

I guess I'm confused; why would you, as an admin, want to allow your users to use less secure passwords? I get that your argument is that all traffic is local and so complex passwords are unnecessary (debatable, I would hold that a strong password policy would still be desirable for defense-in-depth, depending on the network and risk profile), but that doesn't make them undesirable.

We were in the middle of setting everything up company wise (for just two of us), didn't have the password manager yet and I was dictating the password to them. It might be a stupid use case, I agree, but not unreasonable. I didn't say strong passwords are undesirable I said forcing the admin of self hosted software to have to use strong passwords with not even a --yes-im-stupid flag is undesirable

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#166

Post by rcxdude »

chrisjj wrote: post #136
rcxdude wrote: post #127

Earlier quoted context omitted.

If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?

Yes, the fine print. Only this so-called "AI" needs it full size on the front page.

I would consider that more honest than more of a problem.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#167

Post by vel0city »

stonedivot wrote: post #164

This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.

Getting behind the wheel of an F1 car on a track involves lots of proving time that you can actually handle such a vehicle. Meanwhile anyone with a credit card can grant an AI system to impersonate their access as a starting point.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#168

Post by inigyou »

rcxdude wrote: post #127
rrr_oh_man wrote: post #113

Earlier quoted context omitted.

That would be fine if Claude was open source and/or unpaid

If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?

That's actually meaningless in Europe under the Cyber Resiliency Act by the way. They are liable even if they say they aren't.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#169

Post by lonelyParens »

crooked-v wrote: post #160
pllbnk wrote: post #44

Earlier quoted context omitted.

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandbox…

For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.

stealing this

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#170

Post by inigyou »

wredcoll wrote: post #86
jerf wrote: post #21

Earlier quoted context omitted.

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd lik…

I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts! As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.

No need to go back to the Slashdot days. Look no further than 2 days ago to find someone arguing the kernel is uniquely important (here for reliability): https://news.ycombinator.com/item?id=49181366
Post reply on HN