Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

881–890 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#881

Post by caminante »

ses1984 wrote: post #254
caminante wrote: post #229

Earlier quoted context omitted.

The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong." Actual quote: > “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”

Engineering controls basically mean making it impossible to do something in a way that results in catastrophe.

Good point.

My experience is that everyone thinks their defensive controls are air tight until inevitably they're going through a post-mortem on a failure where someone says, "Whelp...Murphy's Law..."

Re: An AI agent deleted our production database. The agent's confession is below

#882

Post by pshirshov »

Recently I've met a guy (a reasonably reach business owner), who confessed me that he really likes to do agentic coding but he doesn't have the expertise, doesn't have enough time and the agents mess up. So he wants to hire a programmer to oversee/replace agents.

Re: An AI agent deleted our production database. The agent's confession is below

#883

Post by caminante »

maxbond wrote: post #243
caminante wrote: post #229

Earlier quoted context omitted.

The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong." Actual quote: > “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”

I'd be interested to hear why my restatement was incorrect. I'm confident that it's what Murphy meant, mostly because I've read his other laws and that's what I recall as the general through line. But that's was a long time ago and perhaps I'm misremembering or was misinterpreting at the time.

Sorry, didn't mean for my comment to come off mean. I can see how it is pedantic or maybe more subjective opinion.

Your phrasing is right.

I was just doing a quick take on this qualifier:

> which is not prevented by a strong engineering control

Re: An AI agent deleted our production database. The agent's confession is below

#884

Post by compass_copium »

HighGoldstein wrote: post #862
grey-area wrote: post #238

Earlier quoted context omitted.

Don’t forget learning, humans can learn, LLMs do not learn, they are trained before use.

Do we? Or are we born with pre-training (all the crucial functions the brain does without us having to learn them) and a context window orders of magnitude larger than an LLM?

It is incredible how willing and eager AI boosters are to denigrate the incredible miracle of human consciousness to make their chatbots seem so special.

No, we are not born with all the pre-training we need. That is rather the point of education, teaching people's brains how to process information in new, maybe unintuitive ways.

Re: An AI agent deleted our production database. The agent's confession is below

#885

Post by veunes »

"backups in the same volume" aren't backups, they’re just snapshots in the same blast radius fwiw. If your DR plan hinges on a single physical volume ID, you have zero resilience

This needs to be a lesson for everyone: real backups belong in an independent store (S3/GCS) in a different region with object lock enabled. It’s the only way to make sure even a compromised root token can’t nuke your data for 30 days

Re: An AI agent deleted our production database. The agent's confession is below

#886

Post by maxbond »

caminante wrote: post #883
maxbond wrote: post #243

Earlier quoted context omitted.

I'd be interested to hear why my restatement was incorrect. I'm confident that it's what Murphy meant, mostly because I've read his other laws and that's what I recall as the general through line. But that's was a long time ago and perhaps I'm misremembering or was misinterpreting at the time.

Sorry, didn't mean for my comment to come off mean. I can see how it is pedantic or maybe more subjective opinion. Your phrasing is right. I was just doing a quick take on this qualifier: > which is not prevented by a strong engineering control

I appreciate it, but it didn't come off as mean and I appreciate the correction. Incidentally Murphy apparently didn't write a whole set of laws so I have no idea what I read to that impression. I did some reading and there are interesting interpretations I hadn't considered that are more pessimistic, which is perhaps what you were flagging. Like that when you add more engineering controls, you create new vulnerabilities, and so things will continue to "go wrong".

If I use this phrasing again I'll present as something derived from or analogous to Murphy's Law rather than a "restatement".

Re: An AI agent deleted our production database. The agent's confession is below

#887

Post by naasking »

> We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete.

So you effectively gave a junior dev a token with the authority to destroy your database, and then complained that the junior dev actually did so by accident while trying to solve some problems it had?

Obviously the AI shouldn't just search everywhere for bearer tokens to try when it runs into a roadblock, but frankly most of the blame does not fall on the AI here IMO. Know what authorities your bearer tokens grant, and understand the consequences of where you store them.

Re: An AI agent deleted our production database. The agent's confession is below

#888

Post by nh2 »

simonh wrote: post #542
MagicMoonlight wrote: post #405

Earlier quoted context omitted.

Actually no, it will increase it. Because it’ll be trained with the deletion command as a valid output.

Exactly. It’s just giving the LLM a token pattern, and it’s designed to reproduce token patterns. That’s all it does. At some point generating a token pattern like that again is literally it’s job.

Why would one set up reinforcement learning like that?

The point of creating samples from user data should surely be to label them good or bad, based on the whole conversation.

You look at what happened eventually, judge the outcome as bad, and thus train the "rm" token in the middle to be less likely.

Re: An AI agent deleted our production database. The agent's confession is below

#889

Post by AtNightWeCode »

jiggawatts wrote: post #880
AtNightWeCode wrote: post #769

Earlier quoted context omitted.

With the difference that best practices in Azure SQL have always been to store your own copies of backups and run the database in some HA/GEO-redundancy mode that blocks deletion.

Which sounds great, except that Azure SQL -- like many cloud services -- was carefully designed to be a tarpit into which you can import your data, but can't get your data back out. For example, for at least a few years its "external" backups were simply the bacpac export function, which wasn't transactionally consistent and had all sorts of fun limits.

Yeah, still some fun limits in Azure SQL. Like you can't take the databases offline or pause the service.

Re: An AI agent deleted our production database. The agent's confession is below

#890

Post by Ntrails »

bombcar wrote: post #322
baal80spam wrote: post #57

Earlier quoted context omitted.

Your dad was a wise man. In my country there is a saying: "Graveyards are full of pedestrians that had the right of way".

“You have the right of way but you can be dead right.”

My fathers different but related saying:

Better to be late than dead on time.

Post reply on HN