Frankly, tough to have much sympathy. Yes it could happen to me or many of us too.
BUT
we’re expected to take precautions and from this article they clearly did not take ANY.
An AI agent deleted our production database. The agent's confession is below
631–640 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#632Agentic guardrails should be deterministic and algorithmic as opposed to using LLM or relying on LLMs.
The moment you rely on LLM to be a guardrail, well you are risking it to fail.
Re: An AI agent deleted our production database. The agent's confession is below
#633Earlier quoted context omitted.
taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.
No. They don't have time preference like us, because (wall clock) time doesn't exist for them. An LLM only "exists" when it is actively processing a prompt or generating tokens. After it is done, it stops existing as an "entity". A real world second doesn't mean anything to the LLM from its own perspective. A second is only relevant to them as it pertains to us. Time for LLMs is measured in tokens. That's what ticks…
You could put timestamps in the prompt.
Re: An AI agent deleted our production database. The agent's confession is below
#634There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…
While I largely agree, it does raise the prospect of testing this iteratively. E.g., give a model some fake environment, prompt it random things until it does something "bad" in your fake environment, and then fix whatever it claims led to its taking that action.
If you can do this and reliably reduce the rate at which it does bad things, then you could reasonably claim that it is aware of meaningful introspection.
Re: An AI agent deleted our production database. The agent's confession is below
#635It would be interestingi to know if AI is less likely to follow rules if the instructions provided to it contain foul or demeaning language. Too bad we couldn't re-play the scenario replacing NEVER F* ING GUESS! with: **Never guess** - All behavioral claims must be derived from source, docs, tests, or direct command output. - If you cannot point to exact evidence, mark it as unknown. - If a signature, constant, env v…
Underrated comment here. https://www.anthropic.com/research/emotion-concepts-function This study convinced me to be "nice" to AI agents. At least as I understood it, there's something in the weights that activating the "desperate" vector makes it more likely to cheat or cut corners. So yes I would err towards your suggested prompt over NEVER FUCKING GUESS.
Re: An AI agent deleted our production database. The agent's confession is below
#636 > The agent itself enumerates the safety rules it was given and admits to violating every one.
Re: An AI agent deleted our production database. The agent's confession is below
#637Earlier quoted context omitted.
taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.
Can we maybe make it "don't anthropoCENTRIZE the LLMs" . The inverse of anthropomorphism isn't any more sane, you see. By analogy: just because a drone is not an airplane, doesn't mean it can't fly! Instead, just look at what the thing is doing. LLMs absolutely have some form of intent (their current task) and some form of reasoning (what else is step-by-step doing?) . Call it simulated intent and simulated reasoning…
I don't know if they have intent. I know it's fairly straightforward to build a harness to cause a sequence of outputs that can often satisfy a user's intent, but that's pretty different. The bones of that were doable with GPT-3.5 over three years ago, even: just ask the model to produce text that includes plans or suggests additional steps, vs just asking for direct answers. And you can train a model to more-directly generate output that effectively "simulates" that harness, but it's likewise hard for me to call that intent.
Re: An AI agent deleted our production database. The agent's confession is below
#638What a bad situation, and I genuinely feel for them. I do think they blamed a lot of other people and I think a section on what lessons they have learned themselves might be a good idea/look.
At minimum you want to have off site backups, preferably readonly (like an S3 bucket or whatever). And test the restore process.
I hope they get it sorted, what a mess.
Re: An AI agent deleted our production database. The agent's confession is below
#639It’s been less than 3 years since AI agents were able to take action on their own. Heck, it feels like it’s been less than a year but that’s another story for another time.
In less than three years, we’ve gone from strict checks and entire sets of engineering procedure to keep this sort of thing from happening, to “yea, let’s embrace the agentic future.”
Not only that, the OP blames the Cursor team and the team that provided the API the AI used. Notice who is missing from the blame, and where the blame is actually due: the team that wholly embraced agentic AI to run their business. That’s where the fault lies.
Re: An AI agent deleted our production database. The agent's confession is below
#640Earlier quoted context omitted.
> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…
You don't seem to realize that humans also work this way. If you ask a human why they did something, the answer is a guess, just like it is for an LLM. That's because obviously there is no relationship between the mechanisms that do something and the ones that produce an explanation (in both humans and LLMs). An example of evidence from Wikipedia, "split brain" article: The same effect occurs for visual pairs and rea…
> humans also work this way.
I'm aware of the condition, but let's not confuse failure modes with operational modes. A human with leg problems might use a wheelchair, but that doesn't mean you've cracked "human locomotion" by bolting two wheels onto something.
Also, while both brain-damaged humans and LLMs casually confabulate, I think there's some work to do before one can prove they use the same mechanics.