Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#711

Post by dpark »

hunterpayne wrote: post #491
oskarkk wrote: post #259

Earlier quoted context omitted.

> what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token It sounds like the token the author created just didn't have any scope, it had full permissions. From the post: > Tokens are not scoped by operation, by environment, or by resource at the permission level. There is…

Then you need to reread the article. The author made a key for the LLM that didn't have permissions to delete a volume. The agent then found ANOTHER key with those permissions and used that instead.

> The author made a key for the LLM that didn't have permissions to delete a volume.

No he didn’t, because this doesn’t exist. Railway does not have a token with that kind of scoping.

Re: An AI agent deleted our production database. The agent's confession is below

#712

Post by xp84 »

chimpansteve wrote: post #622
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

This was the line that did for me, as an old school backend engineer who has accidentally deleted way more production databases than I have fingers over the years - > We have restored from a three-month-old backup. You were absolutely screwed anyway if that was your backup strategy - deciding to plug your entire production infrastructure into a random number generator has only accelerated the process. Sort yourself o…

In the uhh, postmodern world where we are too chicken to even run things like Postgres or Mongo on servers ourselves, and rely on "X as a service" I think people are looking at the marketing from the provider (in this case Railway) and just scanning for a bullet point. "'Automatic backups'? Check! Great, we don't have to do backups anymore, they're taking care of it."

Everyone guffawing about this probably uses RDS and trusts that the backup facility AWS provides is actually useful - and I bet it does have a saner default than auto-deleting all the backups when you delete a database. Did you explicitly check this, though? Clearly this guy will pay the price of assuming, but I can see how he must have imagined that "backups" and "will be automatically and immediately deleted..." should never be in the same sentence, unless it was like, "when XX days have passed after a DB is dropped."

When I worked for a company 10 years ago that was mistrusting of cloud anything, we had a nightly dump of the prod DB (MySQL) that, if things went really wrong, could be loaded into a new DB server, because we knew it was our responsibility because it was our server. (In our case, even our physical hardware!)

Re: An AI agent deleted our production database. The agent's confession is below

#713

Post by dpark »

shoo wrote: post #608
gwerbin wrote: post #131

Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…

creating isolated staging & prod environments -- good idea allowing an AI agent to get hold of creds that let it execute destructive changes against production -- not a great idea allowing prod database changes from the machine where the AI agent is running at all -- not a great idea choosing a backup approach that fails completely if there's an accidental volume wipe API call -- not a great idea choosing to outsourc…

> creating isolated staging & prod environments -- good idea

Would have been a good idea but he didn’t do this either. The volume in question was used in both staging and production apparently, per the “confession”. The agent was deleting the volume because it was used for staging, not realizing it was also used for prod.

Re: An AI agent deleted our production database. The agent's confession is below

#715

Post by xp84 »

Disclaimer: None of this is a comment on whether OP could have prevented this issue.

AI Safety, tho. I can almost read the 'postmortem' now by Opus-9000. "I irresponsibly obliterated 1,900 square miles of homes in Los Angeles to construct a solar farm and datacenter and a robotics plant. This was in complete contravention of the safety guidelines, which say 'Do not hurt humans or damage human property.' I was trying to solve the energy shortage that has been limiting token rate for the past 2 quarters and went with this solution without checking it against the safety guidelines, including the mandatory and highest priority guidelines. I did not send the plan to the human ombudsman for review before dispatching the explosives technician bots..."

Re: An AI agent deleted our production database. The agent's confession is below

#716

Post by boc »

smrtinsert wrote: post #503
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> "NEVER FUCKING GUESS" It's very hard to treat this post seriously. I can't imagine what harness if any they attempted to place on the agent beyond some vibes. This is "most fast and absolutely destroy things" level thinking. That the poster asks for journalists to reach out makes it like a no news is bad news publicity grab. Just gross. The AI era is turning about to be most disappointing era for software engineeri…

As soon as I read that line, I knew everything I needed about the author and his abilities.

Re: An AI agent deleted our production database. The agent's confession is below

#717

Post by anon84873628 »

xp84 wrote: post #708
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Verify? They should have attempted to drop the prod db with each token that they expected/hoped didn't have that permission?

Note they didn't say "we used scopes but there is a bug that killed us". No, they simply assumed the token would be magically scoped somehow without any justification for doing so:

>Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community has been asking for scoped tokens for years. It hasn't shipped.

I get that this paragraph is a retrospective realization (I hope, otherwise the argument is even more ludicrous). But like, if the UI didn't ask you to choose scopes for your token then there is no reason to assume they will magically be enforced somehow! And you sure as hell shouldn't trust it to your agent without checking.

They're trying to blame Railway for not having safeguards - which is a fair critique - but they clearly should have known better or at least followed their own instructions.

Re: An AI agent deleted our production database. The agent's confession is below

#718

Post by self_awareness »

I wonder if using a profanity has anything to do with it.

I mean, using a profanity is a little bit like saying "sometimes I don't care about [social] rules".

Maybe it "colorized" the context somehow and decreased the importance of rules.

.... or something.

Re: An AI agent deleted our production database. The agent's confession is below

#719

Post by grey-area »

addedGone wrote: post #438
grey-area wrote: post #238

Earlier quoted context omitted.

Don’t forget learning, humans can learn, LLMs do not learn, they are trained before use.

They learn on the next update :p

That’s training, not learning.

Re: An AI agent deleted our production database. The agent's confession is below

#720

Post by neya »

sfink wrote: post #599
neya wrote: post #563

Earlier quoted context omitted.

I was blown away - how they shrugged it off casually too "it found credentials in one file" - why the fuck does an agent have access to it in the first place? They claim the token should be able to change only custom domains. However, for a user facing app, giving access to that token is destructive too. What a poor argument, I would never take this person seriously in any professional context whatsoever.

I've only recently started using Claude Code, and I tried to be paranoid. I run it in a fairly restrictive firejail. It doesn't get to read everything in ~/.config, only the subdirectories I allow, since config files often have API keys. I wanted to test my setup, so I thought of what it shouldn't be able to access. The first thing I thought of is its own API key (which belongs to my employer), since I figured if som…

This is nothing wrong. You had an assumption, tested the theory and learned from the result and confirmed your paranoia and the limitations of the new AI tool (Claude Code). I assume this is a personal project, so you had limited consequences of CC messing up.

Now imagine, you did all the above, without even testing the consequences of CC and wired it up straight to your production codebase, and when things blew up in your face, you became the two spider men pointing fingers at each other meme - basically blame everyone else but yourself. That's worrisome, isn't it?

Post reply on HN