Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

221–230 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#221

Post by khazhoux »

pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

> systemic failures across two heavily-marketed vendors that made this not only possible but inevitable.

> No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing.

> The agent that made this call was Cursor running Anthropic's Claude Opus 4.6 — the flagship model. The most capable model in the industry. The most expensive tier. Not Composer, not Cursor's small/fast variant, not a cost-optimized auto-routed model. The flagship.

The tropes, the tropes!!

https://tropes.fyi/

Re: An AI agent deleted our production database. The agent's confession is below

#222

Post by jeremyccrane »

pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

Not some vibe coder, and AI agents can be incredibly powerful. But yes, the irony is not lost on us!

Re: An AI agent deleted our production database. The agent's confession is below

#223

Post by grey-area »

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.

Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundamental understanding of how these word generators work. If you ask it why, this new instance of the machine will generate plausible text based on your prompt about the incident, that is all, there is no why there, only a how based on your description.

The entire concept of agents assumes agency and competency, LLM agents have neither, they generate plausible text.

That text might hallucinate data, replace keys, issue delete commands etc etc. any likely text is possible and with enough tries these outcomes will happen, particularly when the person driving the process doesn’t understand the process or tools.

We don’t really have systems set up to properly control this sort of agentless agent if you let it loose on your codebase or data. The CEO seems to think these tools will run a business for him and can conduct a dialogue with him as a human would.

Re: An AI agent deleted our production database. The agent's confession is below

#224

Post by jeremyccrane »

amluto wrote: post #214
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

Yep.

Re: An AI agent deleted our production database. The agent's confession is below

#225

Post by jeremyccrane »

sieste wrote: post #196
59nadir wrote: post #17

Earlier quoted context omitted.

> a misunderstanding in the users mind about how the agent work On top of that the agent is just doing what the LLM says to do, but somehow Opus is not brought up except as a parenthetical in this post. Sure, Cursor markets safety when they can't provide it but the model was the one that issued the tool call. If people like this think that their data will be safe if they just use the right agent with access to the sa…

Good point, it's like having an instruction "Never fucking output a token just because it's the one most likely to occur next!!1!"

That is actually pretty good, LLM's gonna LLM

Re: An AI agent deleted our production database. The agent's confession is below

#226

Post by [deleted] »

elliotpage wrote: post #202
Quarrelsome wrote: post #197

Earlier quoted context omitted.

its much more aggravating that it looks like they're learning nothing by pushing blame onto everything else except themselves.

I'm glad that I'm not the only person who felt this! It does feel like the post is missing some deserved self-reflection.

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#227

Post by jeremyccrane »

dentemple wrote: post #87
NewsaHackO wrote: post #16

Earlier quoted context omitted.

Twitter users get paid for these 'articles' based on engagement, correct? That may be the reason why it is so dramatized.

It's one way for the company to make its money back, I guess.

Naw, we just want people to know. We followed all Cursor rules, thought we had protected all API keys, and trusted the backups of a heavily used infrastructure company. Cautionary tale sharing with others.

Re: An AI agent deleted our production database. The agent's confession is below

#228

Post by saulpw »

cyclopeanutopia wrote: post #179
pierrekin wrote: post #109

Earlier quoted context omitted.

This is something I really hope can be solved. I long for a “copilot” that can learn from me continuously such that it actually helps if I teach it what I like somehow.

And what will your role be, then?

Teacher.

Re: An AI agent deleted our production database. The agent's confession is below

#229

Post by caminante »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong."

Actual quote:

> “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”

Re: An AI agent deleted our production database. The agent's confession is below

#230

Post by jeremyccrane »

Lionga wrote: post #177
hu3 wrote: post #140

The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.

The most aggravating fact is that the AI slopper that got owned by his dumbness and AI just post an AI generated post that will generate nothing but schadenfreude

AI slopper here :) Kind words from a human. The irony is, there is tremendous truth in the post but you used big words so good for you bud.
Post reply on HN