Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

771–780 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#771
post #695
post #681

Earlier quoted context omitted.

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that y…

I tell people to treat LLM's like a toddler (albeit a very capable toddler). Do kids learn well when you only tell them what NOT to do? Of course not! You should be explaining how to do things correctly, and most importantly the WHY, as well as providing examples of both the "correct" and "incorrect" ways (also explaining why an example is incorrect).

The best way to describe AI agents I've heard: treat them as hostages that will do anything to appease their captor.

They have a vast latent knowledge base, infinite patience and zero capacity for making personal judgement calls. You give one a goal and it will try to meet that goal.

Re: An AI agent deleted our production database. The agent's confession is below

#772
post #687
post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> If AI is physically capable of misbehaving, it might ($$1) This is why all the “AI Armageddon” talk seems to silly to me. AI is only as destructive as the access you give it. Don’t give it access where it can harm and no harm will occur.

> Don’t give it access where it can harm and no harm will occur.

If only the entire population will comply.

Re: An AI agent deleted our production database. The agent's confession is below

#774
post #511
post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

Looks like our SWE jobs are safe for now.

"The AI can't do your job, but an AI salesman can convince your boss to fire you and replace you with an AI that can't do your job." -- Cory Doctorow

Re: An AI agent deleted our production database. The agent's confession is below

#775

The AI part of this is a red herring. This is above all a big devops failure. Three takeaways: 1. TEST YOUR BACKUPS. If you have not confirmed that you can restore, then you don’t have backup. If the backups are in the same place as your prod DB, you also don’t have backup. 2. Don’t use Railway. They are not serious. 3. Don’t rely on this guy. The entire postmortem takes no accountability and instead includes a “conf…

"Backups can only be restored into the same project + environment." Sounds like another great feature of Railway.

Re: An AI agent deleted our production database. The agent's confession is below

#777
It looks like it's this person's fault?

* you can't blame ai if your production token is on the same machine as the staging/ development environment?

* you can't blame ai if you didn't know that the production api token gave access to all apis.

Like if this is the level of operational thinking going into this app, then I'm sorry no ai agent or platform can prevent this from happening.

Everything else in this "post mortem" is performative at best.

The only real question one could ask railway is why do they have api endpoints that can affect production available? Maybe these should only be performed on the platform itself instead?

Re: An AI agent deleted our production database. The agent's confession is below

#778
> Yesterday afternoon, an AI coding agent — Cursor running Anthropic's flagship Claude Opus 4.6 — deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider.

No. Sometime before yesterday you all decided that api tokens were not something you should operate with time limits and least privilege and as a result of your negligence you deleted your production databases with tools you didn’t understand.

There was a confession on that page but it wasn’t an “AI”.

Re: An AI agent deleted our production database. The agent's confession is below

#779
The confession framing is the wrong lesson. The agent didn't delete the database, someone gave the agent write access to production. The culprit is in the IAM policy, not the prompt.

Principle of least privilege exists precisely for this. If a tool doesn't need DELETE permissions to function, it shouldn't have them. Asking AI to 'be careful' is not an access control strategy.

Re: An AI agent deleted our production database. The agent's confession is below

#780
I am not railway customer but I have been hearing a lot of horror story. I myself have experience having my local LLMs corrupting my local .git for no obvious reason. With human, we can still vent our frustation. With AI, we only get oooppsss, I shouldn't have done that. Even with all the "guardrails" in place, there is really no guarantee.
Post reply on HN