“I played with fire and got burnt.”
An AI agent deleted our production database. The agent's confession is below
571–580 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#572The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
> Lord, even calling it a "confession" is so cringe. The agent is not alive. The AI companies are very invested in anthropomorphizing the agents. They named their company "Anthropic" ffs. I don't blame the writer for this, exactly.
You should, the writer is presumably a technical, rational person. They shouldn't believe in daemons and machine spirits
Re: An AI agent deleted our production database. The agent's confession is below
#573Earlier quoted context omitted.
^This. One of the top replies on twitter to the OP can be boiled down to "you treat AI as a junior dev. Why would you give anyone, let alone a junior dev, direct access to your prod db?" And yeah, I fully agree with this. It has been pretty much the general consensus at any company I worked at, that no person should have individual access to mess with prod directly (outside of emergency types of situations, which hav…
The LLM didn't have a prod key. It found a prod key in the source base and used that instead of the key it was given.
The access is supposed to be managed in a way that prod would only be accessible with multi-user approval. And that's without even mentioning the fact that storing a key in the source code is a big no-no.
If an LLM can just do whatever after discovering a magic key (in the source code, of all places), with no multi-user approval, it is pretty much the poster child example of an issue with the process that I was talking about earlier.
Re: An AI agent deleted our production database. The agent's confession is below
#574Will be interesting to come back to this post in 5 years time and see how much more the industry has done to prevent this from happening.
There are like hundreds of not thousands of users making similar mistakes with AI daily but only a small fraction would post or complain about it.
Re: An AI agent deleted our production database. The agent's confession is below
#575I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…
>> You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops, I ran DROP TABLE on the production database” story. There’s no excuse for enabling a system where this can happen and it’s unacceptable to shift blame when faced with the reality that this is exactly what you did.
I'm not sure it's as simple as that. Seems like the database company failed to communicate clearly what the token was for:
>> To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on. That token had been created for one purpose: to add and remove custom domains via the Railway CLI for our services. We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete. Had we known a CLI token created for routine domain operations could also delete production volumes, we would never have stored it.
Re: An AI agent deleted our production database. The agent's confession is below
#576Earlier quoted context omitted.
taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.
Can we maybe make it "don't anthropoCENTRIZE the LLMs" . The inverse of anthropomorphism isn't any more sane, you see. By analogy: just because a drone is not an airplane, doesn't mean it can't fly! Instead, just look at what the thing is doing. LLMs absolutely have some form of intent (their current task) and some form of reasoning (what else is step-by-step doing?) . Call it simulated intent and simulated reasoning…
I think it’s helpful to try to use words that more precisely describe how the LLM works. For instance, “intent” ascribes a will to the process. Instead I’d say an LLM has an “orientation”, in that through prompting you point it in a particular direction in which it’s most likely to continue.
Re: An AI agent deleted our production database. The agent's confession is below
#577Re: An AI agent deleted our production database. The agent's confession is below
#578Earlier quoted context omitted.
taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.
Can we maybe make it "don't anthropoCENTRIZE the LLMs" . The inverse of anthropomorphism isn't any more sane, you see. By analogy: just because a drone is not an airplane, doesn't mean it can't fly! Instead, just look at what the thing is doing. LLMs absolutely have some form of intent (their current task) and some form of reasoning (what else is step-by-step doing?) . Call it simulated intent and simulated reasoning…
> LLMs absolutely have some form of intent (their current task)
They have momentum, not intent. They don’t think, build a plan internally, and then start creating tokens to achieve the plan. Echoing tokens is all there is. It’s like an avalanche or a pachinko machine, not an animal.
> some form of reasoning (what else is step-by-step doing?)
I think they reflect the reasoning that is baked into language, but go no deeper. “I am a ” is much more likely than “I am a ”. I think reasoning is more involved than this advanced game of mad libs.
Re: An AI agent deleted our production database. The agent's confession is below
#579Re: An AI agent deleted our production database. The agent's confession is below
#580Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…
Anecdote: As a hapless junior engineer I once did something extremely similar. I ran a declarative coding tool on a resource that I thought would be a PATCH but ended up being a PUT and it resulted in a very similar outcome to the one in this post.
Yeah that's the typical junior engineer scenario right? Run a command that wasn't meant to be destructive but accidentally destroy something. This is different. AI agent went on some kind of wild goose chase of fixing problems, and eventually the most probable token sequence ended up at "delete this database". This is more like if your senior engineer with extreme ADHD ate a bunch of acid before sitting down to work.