An AI agent deleted our production database. The agent's confession is below

Post by drob518 » Sun, Apr 26, 2026, 7:47 PM UTC

amelius wrote: post #119

maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok. See also hash collisions.

Post by exe34 » Sun, Apr 26, 2026, 7:48 PM UTC

hu3 wrote: post #140

The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.

Post by mmoll » Sun, Apr 26, 2026, 7:48 PM UTC

jmalicki wrote: post #122

pierrekin wrote: post #103

Earlier quoted context omitted.

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.

Post by Lionga » Sun, Apr 26, 2026, 7:49 PM UTC

hu3 wrote: post #140

The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.

Post by cyclopeanutopia » Sun, Apr 26, 2026, 7:49 PM UTC

pierrekin wrote: post #109

heelix wrote: post #98

Man, such a difference between a human whoops and an AI. Had a junior dev hork all environments, when the script they thought worked in nonprod... did not modify an index like they expected, they were quickly able to wipe out everything else in every environment and every data center. It was such a teachable moment. She was my very first hire when I was asked to build a team. Crazy careful with trust, but verify on t…

This is something I really hope can be solved. I long for a “copilot” that can learn from me continuously such that it actually helps if I teach it what I like somehow.