By now it should be common knowledge that telling an LLM not to do something is not a «safeguard». Access control is.
An AI agent deleted our production database. The agent's confession is below
831–840 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#832[dead]
Re: An AI agent deleted our production database. The agent's confession is below
#833Earlier quoted context omitted.
I get what your saying, but this is resonating with me and making me feel for the author: Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail) Cursor AI: Does destructive operation…
Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.
right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.
Re: An AI agent deleted our production database. The agent's confession is below
#834https://archive.is/aBu0T
[dead]
Re: An AI agent deleted our production database. The agent's confession is below
#835> This is not me speculating about agent failure modes. This is the agent on the record, in writing.
> The pattern is clear.
> In our case, the agent didn't just fail safety. It explained, in writing, exactly which safety rules it ignored.
> This isn't a story about one bad agent or one bad API. It's about an entire industry building AI-agent integrations into production infrastructure faster than it's building the safety architecture to make those integrations safe.
Sigh.
Yes, the pattern is very clear. If the author spent less time writing the article than it would take me to read it, why should I even bother?
The agent deleting their prod database is a direct result of this careless "let me just quickly…" attitude.
Re: An AI agent deleted our production database. The agent's confession is below
#836So people are actually allowing LLMs to touch production databases? That is truly nuts.
Re: An AI agent deleted our production database. The agent's confession is below
#837I keep having this conversation with clients. If you want to allow an LLM to delete, create or update data; you need to do this with a human in the loop, and explicit hitl gating against execution; where the agent can't even call the tool without triggering an update on the UI that has to be confirmed (then the confirmation issues the actual tool call).
Re: An AI agent deleted our production database. The agent's confession is below
#838Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…
The stupidity of people sinks to new lows every day. It's astonishing just how ignorant people are of table stakes, basic technological concepts.
You just gave an AI destructive write access to your production environment? Your production DB got dropped? Good. That's not the AI's fault, that's yours, for not having sensible access control policies and not observing principle of least privilege.
Re: An AI agent deleted our production database. The agent's confession is below
#839Stop personifying LLMs. "It Confessed in Writing." No, it wrote some sentences that are congruent with the prior events in the context window. They're not real engineers. Shouting at them is like shouting at a mountain after a landslide. That's not how it works.
Shouting at them is like shouting at your chainsaw after it just chopped off your foot
*you chopped off your own foot by utilising the tool poorly
Re: An AI agent deleted our production database. The agent's confession is below
#840Stop personifying LLMs. "It Confessed in Writing." No, it wrote some sentences that are congruent with the prior events in the context window. They're not real engineers. Shouting at them is like shouting at a mountain after a landslide. That's not how it works.
The personification seems to be at the training level. When I ask an LLM why it did something destructive, the ideal response would be a matter of fact evaluation of the mistakes that I myself have made in setting up the agent and it's environment, and how to prevent it from happening again. Instead the model itself has been trained to apologize and list exactly what it did wrong without any suggestions of how to actually prevent it in the future.