The biggest rule-break was done, not by the agent or infra company, but by the person who gave such elevated authorization (API key) to an autonomous bot.
Isn’t the biggest rule to have working backups with 3-2-1 strategy?
An AI agent deleted our production database. The agent's confession is below
651–660 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#652Re: An AI agent deleted our production database. The agent's confession is below
#653I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…
I partly agree with you but I think there is more here. The fact is that we are currently in a situation in the industry where large amounts of people in large companies are not coding anymore, even told not to code, are being forced to use LLMs are being laid off whether they use them or not because "AI" (and other things, to be sure). I think this is a good thing to be made public. Perhaps, it may give some people pause on escalating the madness, perhaps not. We can certainly criticize this company, sure, but it is naive to think many companies are not barreling down this same path and this sort of thing is a inevitability.
Re: An AI agent deleted our production database. The agent's confession is below
#654Earlier quoted context omitted.
> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…
I agree to the practical part of this, with two nuances: The full data of what's in an LLM's "consciousness" is the conversation context. Just because it isn't hidden, doesn't necessarily mean it doesn't contain information you've overlooked. Asking "why did you do that" won't reveal anything new, but it might surface some amount of relevant information (or it hallucinates, it depends which LLM you're using). "Analys…
> The full data of what's in an LLM's "consciousness" is the conversation context.
No it's not, see research on hiddens states using SAE's and other methods. TBC, I agree with your second point, though I still believe top level OP was reckless and is now doing the businessman's version of throwing the dog under the bus.
Re: An AI agent deleted our production database. The agent's confession is below
#655The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
"A computer can never be held accountable. Therefore a computer must never make a management decision."--IBM training presentation, 1979
Re: An AI agent deleted our production database. The agent's confession is below
#656Earlier quoted context omitted.
I was blown away - how they shrugged it off casually too "it found credentials in one file" - why the fuck does an agent have access to it in the first place? They claim the token should be able to change only custom domains. However, for a user facing app, giving access to that token is destructive too. What a poor argument, I would never take this person seriously in any professional context whatsoever.
I did notice how Claude can start looking outside of working directory. It may scan home directory and find Homebrew token or SSH keys and wipe your GitHub repo.
Yes, it needs to be sandboxed very carefully. It should have no way to access anything outside of the directories you mount in the sandbox.
Re: An AI agent deleted our production database. The agent's confession is below
#657I find these posts hilarious. LLMs are ultimately story generators, and "oops, I DROP'ed our production database" is a common and compelling story. No wonder LLM agents occasionally do this.
Also funny how people (including LLM vendors, like Cursor) think that rules in a system prompt (or custom rules) are real safety measures.
That's why there's tomes of overlapping AGENTS.slop folders and 100K lines of "docslop" and people inventing "memoryslop" systems to reduce this token burden. But the agents can't really distill even a simple instruction like "don't delete prod" because those three words (who knows how many tokens) are the simplest that that expression can get and the ai needs to "reread" that and every other instruction to "proceed according to the instructions". It never learns anything or gets into good habits. It's very clear from these kinds of threads that concepts of "don't" and "do" are not breaking through to the actions the bot performs. It can't connect its own output or its effects with its model context.
Re: An AI agent deleted our production database. The agent's confession is below
#658[dead]
Re: An AI agent deleted our production database. The agent's confession is below
#659I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…
It’s awful. "We had no clue this token had the permission to delete stuff!" - well buddy you issued it without deciding on permissions, it’s your job to assert that. Your latest recoverable backup is three months old? The rule is 3-2-1, you didn’t follow it. Nobody else to blame but yourself. And on and on he rambles…
This is what stood out to me. I've no actual experience operating in this area, but I have been a very grateful user recipient of backups. Anyway, I thought backups were a nightly thing....? Particularly if that data is essentially your business.
Presumably it costs a bit to set up but it surely it's unacceptable not to set it up?
Re: An AI agent deleted our production database. The agent's confession is below
#660The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
I agree with you completely up until this line: > The agent cannot learn from its mistakes. If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again. Yes this is only probabilistic, but so is a human learning from mistakes. They key difference is that for a human this is unlikely to be removed from their memory in a relevant situation, while for an agent it must b…
> If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again.
In my experience, this isn't true. At least with a version or so ago of ChatGPT, I could make it trip on custom word play games, and when called out, it would acknowledge the failure, explain how it failed to follow the rule of the game, then proceed to make the same mistake a couple of sentences later.