Proceeds to post an AI-generated aftermath report.
This only happens to folks who fundamentally don't understand the technology and maybe shouldn't be in positions of deploying and managing software or systems in the first place.
An AI agent deleted our production database. The agent's confession is below
531–540 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#532Earlier quoted context omitted.
> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics. Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material. Barring that, it i…
I have lived about 40 years beneath ceilings and never personally taken a preventative measure. I allow my kids to walk under not only our own ceiling, but other people's ceilings, and I have never asked those people if their ceilings were properly maintained.
I've had a ceiling fall on me once and once to a friend while on vacation. Just because it hasn't happened to you doesn't mean it hasn't happened to other people.
Re: An AI agent deleted our production database. The agent's confession is below
#533Earlier quoted context omitted.
You might as well be asking a tape recorder why it said something. Why are we confusing the situation with non-nonsensical comparisons? There is no internal monologue with which to have introspection (beyond what the AI companies choose to hide as a matter of UX or what have you). There is no "I was feeling upset when I said/did that" unless it's in the context. There is no ghost in the machine that we cannot see bef…
Sperry's experiments makes it quite clear that the comparison is not nonsensical: humans can't reliably tell why we do things either. It is not imbuing AI with anything more to recognise that. Rather pointing out that when we seek to imply the gap is so huge we often overestimate our own abilities.
It is non-sensical because you're simply bringing in comparisons without anything linking the two. You might as well be talking about how oranges, and bicycles think as well as that is just as relevant as how humans think in this discussion.
In fact, talking about "thinking" at all is already the wrong direction to go down when trying to triage an incident like this. "Do not anthropomorphize the lawnmower" applies to AI as much as Larry Ellison.
Re: An AI agent deleted our production database. The agent's confession is below
#534Why so many comments blame the author? If AI is just a tool, just like a database console, would you blame user for entire database loss if he just tried to update a single row in a table?
It's situational.
The blame on how the tool was used and whether this was negligence. If I hit someone with my car because I was looking at my phone, it's not the tools fault. If I hit someone because my brakes failed due to a manufacturing defect, sure blame the tool.
In this situation, the author didn't understand the API key they created. They also likely told the AI it could do a bunch of things (I have claude code ask me before doing anything except read/plan). So I'm sure he turned off some guardrails.
He expects an API to offer an "are you sure?" - it's an API.
He's blaming everyone but himself.
Re: An AI agent deleted our production database. The agent's confession is below
#535I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…
The entire post reads like it was generated via LLM as well.
I like the way the LLM implies that an API call should have a “type DELETE to confirm”. That would make no sense, and no human would ever suggest or want that, I hope.
Re: An AI agent deleted our production database. The agent's confession is below
#536Ultimately, storing secrets on disk was the problem here. Never store secrets on disk. This is software engineering 101. The excuse that "we didn't know the scope of the token's access" is absurd. You knew it was a secret with access to production infrastructure, that's all you need to know. Their provider only having backups on the same volume as the data is also egregious, but definitely downstream of leaking secre…
"Never store secrets on disk." Wait till you learn how that API stores cryptographic material.
What's your point? Obviously, a secure server storing encrypted data on disk in a manner where it is only accessible through a secured API is not what is being discussed here.
Re: An AI agent deleted our production database. The agent's confession is below
#537Earlier quoted context omitted.
For those who might not know the reference: https://simonwillison.net/2024/Sep/17/bryan-cantrill/ : > Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" –…
404 on that link.
A more direct source (possibly the original source?) I know of is a YouTube video entitled "LISA11 - Fork Yeah! The Rise and Development of illumos" which detailed how the Solaris operating system got freed from Oracle after the Sun acquisition.
The whole hour talk is worth a watch, even when passively doing other stuff. It is a neat history of Solaris and its toolchain mixed with the inter-organizational politics.
YouTube link: https://www.youtube.com/watch?v=-zRN7XLCRhc
Direct link to lawnmower quotes (~38.5 minute mark): https://youtu.be/-zRN7XLCRhc&t=2307
Re: An AI agent deleted our production database. The agent's confession is below
#538As someone who uses quite a couple of different AI providers (codex, glm, deepseek, claude premium among others), i've noticed that claude tends to move too fast and execute commands without asking for permission. For example, if i ask a question regarding an implementation decision while it is implementing a plan, it answers (or not) and immediately proceeds to make changes it assumes i want. Other models switch to…
All the models I have used will frequently jump ahead a ton of steps and not verify any of its assumptions. From generating a ton of code output I didn't ask for, to making a ton of assumptions about what I'm working on without appropriate context.
Yeah, /plan is the only way I can work with them now. Too much "helpful" crap I didn't ask for. Having nightmares of former coworkers who would want to refactor 80% of the code base for a 3 line change. AI doesn't subscribe to "if it ain't broke, don't fix it."
Re: An AI agent deleted our production database. The agent's confession is below
#539Earlier quoted context omitted.
He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)
> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…
Why is this getting downvoted? This is exactly what’s going on here. The LLM has no idea why it did what it did. All it has to go on is the content of the session so far. It doesn’t ‘know’ any more than you do. It has no memory of doing anything, only a token file that it’s extending. You could feed that token file so far into a completely different LLM and ask that, and it would also just make up an answer.
Re: An AI agent deleted our production database. The agent's confession is below
#540Ultimately, storing secrets on disk was the problem here. Never store secrets on disk. This is software engineering 101. The excuse that "we didn't know the scope of the token's access" is absurd. You knew it was a secret with access to production infrastructure, that's all you need to know. Their provider only having backups on the same volume as the data is also egregious, but definitely downstream of leaking secre…
[deleted]