It seems like Railway was able to recover the data finally: https://nitter.net/lifeof_jer/status/2048576568109527407#m
An AI agent deleted our production database. The agent's confession is below
891–900 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#892Earlier quoted context omitted.
I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks. Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do it…
He also seems to be lying, he wrote on Twitter the agent was in plan mode. That part has to be exaggerated.
“Plan” vs “execute” modes seem more like suggestions the models _mostly_ follow. I have absolutely had models (Codex and Sonnet/Opus) perform actions in plan mode they should never have been able to take like editing files or starting to work on a plan that was just created.
Re: An AI agent deleted our production database. The agent's confession is below
#893Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
Guys, did you bother checking the poster's profile? https://xcancel.com/lifeof_jer . SEE THE TWEET BELOW. Smells like a ragebait post to me. Also search online for his alleged "PocketOS" company with software for car rental businesses. I couldn't find anything on Google. (Of course, I might be wrong) "The future of SEO is AIO" https://xcancel.com/lifeof_jer/status/2034409722624061772 March 18
Eh, it seems to be real, but all vibe coded.
Re: An AI agent deleted our production database. The agent's confession is below
#894Earlier quoted context omitted.
He also seems to be lying, he wrote on Twitter the agent was in plan mode. That part has to be exaggerated.
I can’t say for sure, but I think Claude’s mode is nothing more than part of the system prompt. I don’t think it actually takes away web request or file write tools. I say this because I could swear I’ve seen Claude go ahead and make some changes even while we’re in plan mode. Web requests certainly, because it can fetch docs and so forth.
You’re not alone, I’ve absolutely seen the same behavior occasionally with Opus in OpenCode where it takes actions it shouldn’t be able to in plan mode.
Re: An AI agent deleted our production database. The agent's confession is below
#895Earlier quoted context omitted.
Exactly. It’s just giving the LLM a token pattern, and it’s designed to reproduce token patterns. That’s all it does. At some point generating a token pattern like that again is literally it’s job.
Why would one set up reinforcement learning like that? The point of creating samples from user data should surely be to label them good or bad, based on the whole conversation. You look at what happened eventually, judge the outcome as bad, and thus train the "rm" token in the middle to be less likely.
It is possible, but it requires specifically labelling the data. You have to craft question response pairs to label. But even then the result is only probabilistic.
The LLM in this case had been very thoroughly trained and instructed quite specifically not to do many of the things it actually then when off and did.
It may be that there's a kind of cascade effect going on here. Possibly once the LLM breaks one rule it's supposed to follow, this sets it off on a pattern of rule violations. After all what constitutes a rule violation is there in the training set, it is a type of token stream the LLM has been trained on. It could be the LLM switches into a kind of black hat mode once it's violated a protocol that leads it down a path of persistently violating protocols, and given the statistical model some violations of protocol are always possible.
My mother was a primary school teacher. She used to say that the worst thing you can say to a bunch of kind leaving class down the hall is "don't run in the hall". It puts it in their minds. You need to say "Please walk in the hall", then they'll do it.
Re: An AI agent deleted our production database. The agent's confession is below
#896> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…
This is what I am seeing more and more of, both in tech online and in the minds of people around me. Despite peoples' innate curiosity of how LLMs work, they still don't understand at the end of the day that they are just models. Augmented with tools and more capable than ever, yes, but still a piece of math at the end of the day. To expect of it anything other than credible output is science fiction.
Re: An AI agent deleted our production database. The agent's confession is below
#897Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
Guys, did you bother checking the poster's profile? https://xcancel.com/lifeof_jer . SEE THE TWEET BELOW. Smells like a ragebait post to me. Also search online for his alleged "PocketOS" company with software for car rental businesses. I couldn't find anything on Google. (Of course, I might be wrong) "The future of SEO is AIO" https://xcancel.com/lifeof_jer/status/2034409722624061772 March 18
There seems to be quite a lot of stuff here [1]
Seems legit to me. The oldest news item is from 2021. The domain name is new, but there seems to have been some rebranding lately. The product used to be called Pocket RentalOS and even that seems to be fairly recent rebranding [2]
[1] https://pocketos.ai/ [2] https://pocketos.ai/news/pocket-rebrands-its-luxury-rental-m...
Re: An AI agent deleted our production database. The agent's confession is below
#898Earlier quoted context omitted.
Engineering controls basically mean making it impossible to do something in a way that results in catastrophe.
Good point. My experience is that everyone thinks their defensive controls are air tight until inevitably they're going through a post-mortem on a failure where someone says, "Whelp...Murphy's Law..."
Pushing buggy software that could result in some expected nonzero number of incidents per year can be done as an intentional tradeoff, any time the cost of incidents is lower than the opportunity cost of moving fast.
Dare I say that most software engineers literally plan to hit Murphy's Law?
If you build websites, and you never get hit by Murphy's Law, it could mean you are being too conservative.
If you build bridges, your job is to make sure you never get hit by Murphy's Law.
Re: An AI agent deleted our production database. The agent's confession is below
#899Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
I read this as "the agent should have asked for confirmation before running".
Me too. The line before the curl command says the agent ran the command, so it could be that the next paragraph is complaining that the agent didn't ask for confirmation.
Re: An AI agent deleted our production database. The agent's confession is below
#900Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
[dead]