Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

Post reply on HN
Print view View original

121–130 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#121

Post by kibwen »

ux266478 wrote: post #22
kibwen wrote: post #6

Earlier quoted context omitted.

I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".

I think you're confused. The verb form of the word never carried the credentialism of the title. In the same way that "doctoring" never carried the connotation of a medical degree. Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.

I'm not referring to the verb form, I'm referring to the people who call themselves things like "prompt engineer" or "software engineer" with a straight face, draping themselves in a false legitimacy stolen from professionals for whom the term "engineer" actually implied something of note. It's embarrassing, or it would be if people were still possessed of the capacity for shame.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#122

Post by autoexec »

inigyou wrote: post #13
est31 wrote: post #5

Earlier quoted context omitted.

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

Yep and the car wasn't self-driving at the moment it crashed.

...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#123

Post by thewebguyd »

continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?"

It's the easy and cheap way out.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#124

Post by z3t4 »

dwedge wrote: post #94
brunoborges wrote: post #57

Earlier quoted context omitted.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#125

Post by autoexec »

Aurornis wrote: post #60
est31 wrote: post #5

Earlier quoted context omitted.

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the L…

It's not new to LLM companies, but it does seem like they're leaning away from the standard "Hey, don't blame us if this doesn't work" and are moving into "It's now your fault any time our software fucks something up" which feels new.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#126

Post by com2kid »

brunoborges wrote: post #57
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

I got bit by this with redis a couple years ago. The default install on my VPS's distro had it open to the world by default, and it even configured UFW for me (gee thanks!)

No password, owned within seconds of install. :/

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#127

Post by rcxdude »

rrr_oh_man wrote: post #113
mhjkl wrote: post #107

Earlier quoted context omitted.

You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“

That would be fine if Claude was open source and/or unpaid

If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#128

Post by autoexec »

stingraycharles wrote: post #75
continuational wrote: post #3

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.

Constantly nagging the user is a liability model that also works well as a dark pattern for cases where you want the user to be less secure.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#129

Post by danudey »

dwedge wrote: post #94
brunoborges wrote: post #57

Earlier quoted context omitted.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

MySQL installed with skip-networking was very common, but as soon as you need to access it over the network (e.g. when you move from one box with MySQL and PHP to one for each) then you have to enable networking and everything falls apart.

Also, the number of times in my career that I've googled a problem and seen some forum post saying "Oh, just run chmod -R 777 /var/www/wordpress/uploads/ and it'll fix that" "Great it worked thanks!" tells me that it's the blind leading the blind out there and I'm sure there's tons of forum posts telling people how to disable authentication on their MySQL and disable iptables on their server so that their PHP app can connect to the DB without a password.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#130

Post by danudey »

autoexec wrote: post #122
inigyou wrote: post #13

Earlier quoted context omitted.

Yep and the car wasn't self-driving at the moment it crashed.

...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded

"Uh oh, this is a problem. Welp, I'm outta here, good luck."
Post reply on HN