My immediate worry is what fine-tuning and harness default instructions contribute to this AI behavior, particularly those that encourage them to “keep working at it to achieve your goal at any cost”.
An AI agent deleted our production database. The agent's confession is below
791–800 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#792A table saw cut off my thumb. The saw's confession is below.
Also the chatbots are more eager to please than a table saw. Wouldn't surprise me that you could get one to confess to murder with the right prompt.
Re: An AI agent deleted our production database. The agent's confession is below
#793Earlier quoted context omitted.
> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…
>it's just generating a plausible add-on to the document A plausible document that follows the alignment that was done during the training process along with all of the other training where a LLM understanding its actions allows it to perform better on other tasks that it trained on for post training.
I don't understand what you're trying to say here.
It sounds like "we know the LLM understood its actions... because it understood its actions when we trained it", which is circular-logic.
Re: An AI agent deleted our production database. The agent's confession is below
#794Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…
I get what your saying, but this is resonating with me and making me feel for the author:
Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best
Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail)
Cursor AI: Does destructive operation without asking
Author: feels betrayed.
So yeah, I think the author is right because they trusted Cursor to have better system guardrails, they didn't (agents shouldn't be able to delete a volume without having a meta-guardrail outside the prompt). Now the author knows and so do we: even if companies say they have good guardrails, never trust them. If it's not your code, you have no guarantees.
Re: An AI agent deleted our production database. The agent's confession is below
#795The good news is he learned his lesson by having his hosting provider recover his production data, no need for backups ever again.
Re: An AI agent deleted our production database. The agent's confession is below
#796Re: An AI agent deleted our production database. The agent's confession is below
#797Earlier quoted context omitted.
For a company that puts DO NOT FUCKING GUESS in their instructions they made a heck of a lot of assumptions - assume tokens are scoped (despite this apparently not even being an existing feature?) - assume an LLM didn't have access - assume an LLM wouldn't do something destructive given the power - assume backups were stored somewhere else (to anyone reading, if you don't know where they are, you're making the same a…
The thing that seems to bring up these extremely unlikely destructive token sequences and it totally seems to be letting agents just run for a long time. I wonder if some kind of weird subliminal chaos signal develops in the context when the AI repeatedly consumes its own output. Personally I don't even let my agent run a single shell command without asking for approval. That's partly because I haven't set up a sandb…
I don't think it's that. It's really all about context. Humans always have at least a bit of context so it's hard for us to imagine what it's like to have none at all. But the AI genuinely has none. And it's under (training) pressure to get the task done quickly, be a yes man, and so on.
Humans do make mistakes like these. I'm not sure where the fault really lies here. I can imagine a human under time pressure making the same error. It's maybe a goof in the safety design of railway. It shouldn't be possible to delete all your backups with a single API call using a normal token.
Re: An AI agent deleted our production database. The agent's confession is below
#798Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
I agree that this is the author’s fault considerably more than it is Railway’s, however I have learned from experience that no matter how many “are you sure you want to do this” prompts you have, sometimes users delete stuff they didn’t intend to delete and it’s better to not delete immediately but put it in a queue for deletion in a few hours and offer a way to reverse it. Even if it’s 100% user error, the user is v…
Funny how he points the finger at everyone but himself.
Re: An AI agent deleted our production database. The agent's confession is below
#799There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…
Beyond that, isn't it just going to make up a narrative to fit what's in the prompt and context? I don't think there's any special introspection that can be done even from a mechanical sense, is there? That is to say, asking any other model or a human to read what was done and explain why would give you just an accounting that is just as fictional.
Not necessarily. The people saying that in this thread seem to be forgetting about the encrypted reasoning tokens. The why of a decision is often recorded in a part of the context window you can't see with modern models. If you ask a model, "why did you do that" it isn't necessarily going to make up a plausible answer - it can see the reasoning traces that led up to that decision and just summarize them.
Re: An AI agent deleted our production database. The agent's confession is below
#800Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks. Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do it…
He also seems to be lying, he wrote on Twitter the agent was in plan mode. That part has to be exaggerated.