An AI agent deleted our production database. The agent's confession is below

Post by throwdbaaway » Sun, Apr 26, 2026, 9:57 PM UTC

rahoulb wrote: post #12

heliumtera wrote: post #7

Someone trusted prod database to an llm and db got deleted. This person should never be trusted with computers ever again for being illiterate

If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too. The LLM broke the safety rules it had been given (…

    curl -X POST https://backboard.railway.app/graphql/v2 \
      -H "Authorization: Bearer [token]" \
      -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}'

Post by Someone1234 » Sun, Apr 26, 2026, 9:59 PM UTC

jeremyccrane wrote: post #232

easton wrote: post #79

Earlier quoted context omitted.

AWS actually has a thingy on some services called “deletion protection” to prevent automation from accidentally wiping resources the user didn’t want it to (you set the bit, and then you need to make a separate api request to flip the bit back before continuing). I think it’s designed for things like Terraform or CloudFormation where you might not realize the state machine decided your database needed to be replaced…

This should be the solution. All destructive actions require human intervention.

Post by XenophileJKO » Sun, Apr 26, 2026, 9:59 PM UTC

keeda wrote: post #290

xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

Post by pessimizer » Sun, Apr 26, 2026, 10:02 PM UTC

gigatree wrote: post #333

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

Post by 9dev » Sun, Apr 26, 2026, 10:02 PM UTC

Sankozi wrote: post #279

grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

I have opposite view - LLMs have many similarities with humans. Human, especially poorly trained one, could have made the same mistake. Human after amnesia could have found similar reasons to that LLM. While LLM generate "plausible text" humans just generate "plausible thoughts".