Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

191–200 of 329 posts

Re: AI didn't delete your database, you did

#191

Post by JamesSwift »

figmert wrote: post #41
oersted wrote: post #30

Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…

> The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by deleting the volume. But it’s likely that they are somewhat downplaying their culpability by being vague. I was just talking to my girlfriend saying I've realised that I've not written a single line of code, nor have I debugged myself for at least the past 3 mo…

Give an agent an obstacle and it will try to find a way around it. Most of the egregious commands Ive seen it run were fundamentally due to something blocking it from accomplishing a task. So eg if you block network access for the agent, you will get all sorts of creative solutions to try and get around the problem. This is also why its nearly impossible to corral commands. Because eventually it will rot13 encode a script and run it anyways.

Re: AI didn't delete your database, you did

#192

Post by clickety_clack »

BadBadJellyBean wrote: post #145
pjc50 wrote: post #124

Earlier quoted context omitted.

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else. I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives its…

I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces…

A band saw is always a screaming band of bladed death. An LLM is sometimes a buddy, sometimes a mentor, and only sometimes a guy that drops your database.

Re: AI didn't delete your database, you did

#195

Post by pc86 »

kennywinker wrote: post #110
pc86 wrote: post #46

Earlier quoted context omitted.

It seems closer to "roll two or three successive 1s on a D100 and have your LLM hooked directly into your production systems and have your LLM user have DELETE permissions" and probably 1 or 2 other things I'm forgetting.

It pulled an api key from an unrelated file. It wasn’t given delete permission, it found it. Not picking on you specifically, but in general the comments here have me wondering if AI has stolen our basic reading comprehension, or if we were always this bad. Anyway, take “LLM user had delete permission” off your list and add “deleting the production db also deletes all the backups” to the list.

Fair criticism mate. I'll only say that if your backups aren't in a completely separate system you don't really have backups.

Re: AI didn't delete your database, you did

#196

Post by lowbloodsugar »

yubblegum wrote: post #120
lowbloodsugar wrote: post #64

Earlier quoted context omitted.

Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.

> Humans aren’t any better We're different. People have fairly consistent faults. LLMs are nondeterministic even in terms of how they fail. A high value human resource can be counted on to deliver. That, imho, is in fact one of the primary roles of good management: putting the right person in the appropriate position. Process engineering has worked to date because both the human and mechanical components of a system…

> A high value human resource can be counted on to deliver.

Anyone who has encountered politics, psychopaths and narcissists knows that this isn’t always true.

Re: AI didn't delete your database, you did

#197

Post by lowbloodsugar »

jmalicki wrote: post #102
lowbloodsugar wrote: post #64

Earlier quoted context omitted.

Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.

Formal logic AI systems have existed and were popular in the 1980s. One of the problems is that they don't work - in the real world there are no firm facts, everything is squishy, and when you try to build a large system you end up making tons of exceptions for special cases until it becomes completely untenable. Non-deterministic systems that work probabilistically are just superior in function to that, even if it m…

Actually they do very well at medical diagnosis but the doctors union banned them.

Re: AI didn't delete your database, you did

#198

Post by edot »

engeljohnb wrote: post #186
edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

Interns are human. Humans can always be held accountable. A computer never can. Therefore, no one should leave a computer in charge of human decisions.

Exactly. Thus the blame when an LLM does something dumb should fall on the human who owns the implementation of said LLM. A dead simple example: if I paste confidential information into ChatGPT, that’s on me. If I let Codex have access to an environment where it can get to confidential information, that’s also on me. At best I could also blame my IT department for giving me technical permissions to do such a thing, but still it’s humans at fault (and I believe in taking Extreme Ownership, so I wouldn’t even do that). LLMs are just technology like any other.

Re: AI didn't delete your database, you did

#199

Post by program_whiz »

AmbroseBierce wrote: post #175
edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

Well, a significant difference is nobody is selling the concept of interns as the end-all solution of humanity's problems, unlike AI.

Yeah the usual mott and bailey. Monday -- AI is taking over the world, tremble in fear! Tuesday -- sure it did a boneheaded thing, its just a tool, no better than an intern, actually its _your_ fault, all the data in the entire world isn't enough to train this system not to delete prod!

Re: AI didn't delete your database, you did

#200

Post by rayruizhiliao »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

leilani's work is super interesting
Post reply on HN