Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

51–60 of 329 posts

Re: AI didn't delete your database, you did

#51

Post by amluto »

stingraycharles wrote: post #35
edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

> This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the faul…

There is a major issue with current AI tools that they want to effectively grant access to everything their user has access to. The whole sandbox structure is wrong (although various people have vibe coded assorted improvements).

Re: AI didn't delete your database, you did

#52

Post by nimbius »

anyone with twenty years of devops experience is likely to abhor Diallo's hot take and for good reason.

AI is being sold as a developer, as it is being sold as the do-everything alternative to traditional processes and methods. it is not being sold as an intern or a junior, but a real developer.

turning the tables and gaslighting devops professionals into believing the issue isnt an emerging technology with overwhelmingly heavy handed marketing and profitless operating strategy thats been shoehorned into seemingly everything and promises anything, but somehow their own oversight, will destroy whatever "vibe code" market you think you have at the cusp of a global recession.

had this AI been a real programmer chances are great they would have (intelligently) foreseen the possibility of damaging a production environment and asked for help.

to play devils advocate: you could hire a junior dev for a fourth of whatever the AI token spend is, and have likely avoided this issue entirely. sure, a greybeard is going to need to pull themselves away from some fierce sorting algorithm challenge for a second to give a wisened nod, but you would have saved yourself an inexorable amount of headache and profit loss in the longer run.

Re: AI didn't delete your database, you did

#53

Post by BadBadJellyBean »

mock-possum wrote: post #31
newsoftheday wrote: post #13

The article is dumb, "why do you have an API endpoint that deletes your entire production database?" irrelevant, the AI did what it did, period.

No, the AI did what you told it to do. The AI didn’t do anything on its own. > if you're going to use AI extensively, build a process where competent developers use it as a tool to augment their work, not a way to avoid accountability

> No, the AI did what you told it to do.

I'd say yes and no. The LLM reacted to the input that was given but it is not possible for a human (especially without access to the weights) to even guess what will happen after that.

Regardless of that I agree that it's completely the fault of the user to use a tool where you can't predict the outcome and give it such broad permissions and not having a solid backup strategy.

Either don't use non deterministic tools or protect yourself from the potential fallout.

Re: AI didn't delete your database, you did

#54

Post by Steve16384 »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

It's taking "computer says no" to the next level. Computers do exactly what they're told, but who told them? The person entering data? The original programmer or designer of the system? The author of whatever language text was used to feed the ai? Even before AI, it was very difficult to determine who is accountable, and now it's even more obfuscated.

Re: AI didn't delete your database, you did

#55

Post by saghm »

tantalor wrote: post #22
pengaru wrote: post #5

wiring up an RNG to your CLI has fairly obvious risks, the root of the problem is ~everyone's treating GenAI as if it's AGI - the rest is popcorn fodder.

New rule: Roll a 1 on a D20 -> you accidentally delete your own database

This is actually a fun way to describe it. I've being saying for a little while now that using AI for things where there's consequences if it fails is a bad idea, but it never occurred to me that this is basically the same concept as some rules in tabletop RPGs.

In D&D 3.5 edition, there was a rule about how you could "take 20" on a d20 roll to get a guaranteed 20 by taking 20 times as long in-game to perform the action, but only if it was a check that didn't have consequences for failure, since it was essentially a shortcut to skip the RNG of rolling until you rolled a 20. Maybe framing it like this might make sense to people a bit more, but if not, I'll at least have more fun making my case.

Re: AI didn't delete your database, you did

#56

Post by CodesInChaos »

oersted wrote: post #30

Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…

> It turns out too that Railway stores backups in the same volume.

That's probably not quite correct. I'd guess the snapshots are synchronized elsewhere (e.g. object storage). But the snapshots are logically owned by the volume resource, and deleting the volume deletes the associated snapshots as well. I think AWS EBS volumes behave like that as well.

Re: AI didn't delete your database, you did

#57

Post by paroneayea »

spiderfarmer wrote: post #43
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.

The point is not primarily the court. The court is an example of someplace where we have accountability, but we build accountability mechanisms as foundational to most of our computing.

Tracebacks, debuggers, logging, etc. We put enormous resources into not only the bad case, but the potential that a bad case could occur. When something goes wrong, we want to know why, and we want to make sure that something bad like that doesn't happen again.

Re: AI didn't delete your database, you did

#58

Post by mattgreenrocks »

The most exasperating thing about the incident is how much of the media either tried to pin it on AI and/or Railway. The whole thing only took place because the guy FAFO’d by having AI work with prod directly.

Yet the narrative was mostly not about accountability for him. If I was a dumbass and deleted prod and wrote a post about it, nobody would care. Put an AI in there and all of the sudden it’s newsworthy. Ridiculous.

Re: AI didn't delete your database, you did

#59

Post by georgemcbay »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

> so the best thing we can do in their stead is hold them accountable

We can't even do this. They are worth too much money already to ever be held really accountable.

The best we can ever hope for is they might occasionally be hit with relatively insignificant "cost of doing business" fines from time to time.

Re: AI didn't delete your database, you did

#60

Post by avidiax »

spiderfarmer wrote: post #43
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.

The court is the regulator of last resort. A company that gets taken to court would likely have been sanctioned by the government regulators of another country.

Also, court is unavailable in many cases now. Binding arbitration is very common now, but this would be illegal in many other places.

Post reply on HN