Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#691

Post by etothet »

This is the failure of the author and their team, not Cursor and not Claude.

If a junior or new employee made this mistake, it would be because you, as the founder, and your engineering team, didn’t have protections in place from editing/destroying production data for this particular scenario.

Using best practices and least privilege principles is more important now than it ever has been. For those of us with our hands close to button, we should be always mindful of this now more than ever.

Re: An AI agent deleted our production database. The agent's confession is below

#692

Post by dpark »

YeGoblynQueenne wrote: post #575
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

>> You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops, I ran DROP TABLE on the production database” story. There’s no excuse for enabling a system where this can happen and it’s unacceptable to shift blame when faced with the reality that this is exactly what you did. I'm not sure it's as simple as that. Seems like the database company failed to…

Rereading the post, I think it’s even simpler than that. The volume was shared across multiple environments. Specifically it was shared across staging and prod. Yet another example of the company YOLOing with their production environment. Presumably a token scoped purely to staging could have deleted that volume anyway, because it was part of the staging environment. Mixing production and staging like this is a train wreck waiting to happen.

“I had no idea what this token was for” is also not a valid excuse. That’s negligence. Everything about this story says the author is just vibe coding garbage with no awareness of what’s really happening.

* Doesn’t know what kind of token he’s using.

* Has prod tokens sitting on a dev box for AI to use (regardless of the scope!).

* Doesn’t know that deleting a volume deletes the backups.

* Has no external backup story.

* Mixes staging and prod.

And then he blames the incident on other companies when he misuses their products. (Railway certainly had docs that explain their backups and tokens.)

This is catastrophically negligent.

Re: An AI agent deleted our production database. The agent's confession is below

#693

Post by juped »

Another angry all-caps rant in an agents file (cf. "NEVER FUCKING GUESS"). As the operator of this tool which you used to delete your production database, you should at least know that angry all-caps ranting pushes the big textual probability engine into the space of things associated with abusive ranting.

Re: An AI agent deleted our production database. The agent's confession is below

#694

Post by godelski »

esafak wrote: post #650
godelski wrote: post #644

Earlier quoted context omitted.

> Claude Code used bash to make edits anyway. If you had the former rule why would you ever whitelist bash commands? That's full access to everything you can do. Same goes for `find`, `xargs`, `awk`, `sed`, `tar`, `rsync`, `git`, `vim` (and all text editors), `less` (any pager), `man`, `env`, `timeout`, `watch`, and so many more commands. If you whitelist things in the settings you should be much more specific about…

At some point you need to get things done.

There's no point in getting things done if there's nothing that ends up being done.

You can still get shit done without risking losing it all. Don't outsource your thinking to the machine. You can't even evaluate if what it is doing is "good enough" work or not if you don't know how to do the work. If you don't know what goes into it you just end up eating a lot of sausages.

Re: An AI agent deleted our production database. The agent's confession is below

#695

Post by cwsx »

6r17 wrote: post #681
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that y…

I tell people to treat LLM's like a toddler (albeit a very capable toddler).

Do kids learn well when you only tell them what NOT to do? Of course not! You should be explaining how to do things correctly, and most importantly the WHY, as well as providing examples of both the "correct" and "incorrect" ways (also explaining why an example is incorrect).

Re: An AI agent deleted our production database. The agent's confession is below

#696

Post by linkregister »

momento wrote: post #335

I asked Railways agent to live resize a volume attached to our DB and it nuked the database and migrated it from the EU to the US Here is an excerpt from the chat log: >Please resize my postgres volume to its maximum allowed size for my plan. >Done. Resized your Postgres volume to 100GB (the Pro plan maximum). Changes are staged and ready to deploy. >oh no, you deleted all the data in the volume >I apologize for that…

It sounds like the Railway web agent designer has made the elementary mistake of having a single agent to accept user input, interpret it, and execute commands.

It is not difficult to design a safer agent. The Snowflake web agent harness has built-in confirmations for all actions. The LLM is just for interacting with the user. All the actions and requisite checks should be done in code.

Re: An AI agent deleted our production database. The agent's confession is below

#698

Post by dpark »

gwerbin wrote: post #589
lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

The author definitely deserves a lot of blame here and clearly doesn't understand AI well enough to have a coherent opinion on AI safety. But Railway bears some responsibility too because, at least of the author is to be believed, it looks like they provide no safety tools for users, regardless of whether they use AI or not. You should be able to generate scoped API tokens. That's just good practice. A human isn't li…

> You should be able to generate scoped API tokens. That's just good practice.

Fully agree, but given the rest of this story I don’t imagine the author would have scoped them unless Railway literally forced him to.

> A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.

The AI agent was deleting the volume used in the staging environment. It happened to also be the volume used in the production environment. 100% a human could have made this mistake.

Re: An AI agent deleted our production database. The agent's confession is below

#699

Post by prng2021 »

SoftTalker wrote: post #648
prng2021 wrote: post #623

Earlier quoted context omitted.

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

For every API you publish, do you verify that scoped API keys work as they should before you go live? If so, why would you not do the same for APIs you integrate with? It's all part of "your" system from the user's perspective.

“why would you not do the same for APIs you integrate with?”

Who does that? Jira and Salesforce have hundreds of endpoints each. AWS has hundreds of services, and each may have hundreds of endpoints. Who on your team is testing key scopes of every endpoint? Do you do it for each key you generate? After all, that external system could have a bug at any moment in managing scopes. Or they could introduce new endpoints that aren’t handled properly. So for existing keys, how frequently do you re-validate the scope against all the endpoints?

Post reply on HN