Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

931–940 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#931

Learn to code yourself, stop using slop generators, then shit like this doesn't happen.

Senior software dev brother :)

it reads as very mid-level - enough technical depth to identify problems, but not enough to know where to focus. The major point of writing post mortem documentation is to identify your own flaws and risks that led to the issue, so you can fix your own stuff, not to throw a list of action items over the fence. you especially do not write somebody else a bunch of action items without getting their review before publishing.

first off, you are building and running a DBA agent in production, so as a reviewer I want to know why the deployment pipeline for your agent didn't catch this error. What test are you missing? How are you going to improve the test harness for the future?

Id also want to hear about industry best practices, based on comments in this thread, "NEVER FUCKING GUESS" is a prompting anti-pattern that creates more desperate outputs to get the calls done, but id expect your prompt to have a line for output formatting like "this operation cannot be completed with the given api key"

there are also dev ops best practices - you should be deploying your db changes like you deploy code, with code review. You should have a really good reason to skip running db migrations through a deployment pipeline with appropriate tests all the way through, to instead use your dba agent separately for each stage. Its pretty standard that teams use agents to produce deterministic code, then deploy that; thats a simple process change that would mitigate most of the deleting prod risk. Did your changes to production follow something like a 2 person review? have two people look at the commands to run before running them? why not?

the agent response accurately points out a risk which goes unaddressed - why do you have staging and prod commingled? Have you fixed that problem yet by making a second account or volume or whatever that gives you stage isolation? if you are purposefully having staging run against the prod tables, staging is prod

a senior post mortem should be clearly actionable by your own team to make that not happen again. You own your system, not cursor or railway. Maybe you considered these things in a different document, but the only other thing you point at is that you first wanted to blame anthropic.

Re: An AI agent deleted our production database. The agent's confession is below

#932

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

He didn't give it root access, it found root access.

Re: An AI agent deleted our production database. The agent's confession is below

#933
post #930

Earlier quoted context omitted.

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

Just wanted to say that I share any frustration you may feel at every reply to your comment completely missing the point

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#934
post #58

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

User is an idiot for using AI Agent. But I am not saying that it is not also badly designed system. Soft delete or something like should be standard for this type of operations. And any operator should know well enough to enable it for production.

They don't "know" anything is the point - they're trying to complete a task and often get confused while doing so. Until reliability of task completion approaches several 9's, which we're a long way off from, this is always going to be a thing.

Re: An AI agent deleted our production database. The agent's confession is below

#935

Earlier quoted context omitted.

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

Here is what they say, at the very top they explain that llm's are inherently unreliable. It looks like they offer security tools and safeguards, but they also provide an auto run option. There is nothing a vendor can really be responsible for someone shooting themselves in the face. You can argue that they shouldn't provide that, but that's what people want, so they do, with warnings.

It sounds like this user either didn't use security controls, approved prompts they didn't understand, or disabled the checks entirely. Working in IT/tech a big chunk of my life so far and seeing all the dumb crap people who even know better do, I would bet my house on that being the most likely scenario rather than cursor somehow being at fault here.

https://cursor.com/docs/enterprise/llm-safety-and-controls

Re: An AI agent deleted our production database. The agent's confession is below

#936
post #923

Earlier quoted context omitted.

right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

> At some point you have to trust vendors. You absolutely do not. When someone makes an unbelievable claim, such as having magic guardrails for LLMs that prevent dangerous actions (what would that even mean?!), you don’t have to trust that claim. If you trust someone’s claim without justification, that’s on you.

They don't even claim this:

https://cursor.com/docs/enterprise/llm-safety-and-controls

Re: An AI agent deleted our production database. The agent's confession is below

#937
post #922

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

> Guy gives non-deterministic software root access, desaster happens. I agree the guy is an idiot for trusting these AI models. OTOH AI companies keep running and marketing their services with zero accountability for mistakes.

I wonder, how should an AI company be accountable for non-deterministic nature of AI, which is a fundamental property of the said AI?

People have been drinking too much hopium they have lost touch with reality.

Everyone needs to properly understand these tools before they use them for anything serious.

Re: An AI agent deleted our production database. The agent's confession is below

#938

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

It's not common, but I've personally built APIs where requests for dangerous modifications like this perform a dry run, giving in the response the resources that would be deleted/changed and a random token, which then needs to be provide to actually make the change. The idea was that this would be presented in the UI for the user to confirm, but it should be as useful or more by AI agents. Also, you get the benefit t…

I tested a similar approach, but the issue, along with the solution to that issue, is that they’re autocomplete engines. Phrases like “Reply X to confirm” are a request with a high probability that X becomes the response. If you zoom out and look at the sequence from a text continuation perspective, once the ‘delete’ tokens are in play the “confirm” step is just how that exchange tends to go. It’s a bit like saying “Begin your response by saying ‘Yes’, then decide if that’s really the case.”

But you can simulate the effect of thinking and shift the token probabilities around by gaslighting it and having it explain the effect of running the command before it does it. What I found worked well was when a destructive command was detected my system automatically ignored it and edited the prior message to tack on a variation of “Briefly step through the effect of {{command}}, then continue the task.” It has ‘no idea’ why it’s explaining the command, as far as it ‘knows’ it didn’t issue the command and thus it’s not committed to a probability sequence that ends with confirming it. However, if the explanation includes “it would destroy the production database” then the continuation tends not to lead to issuing the command. But if it came through a second time it was allowed to run.

I quit bothering with it when I found that ‘destructive typos’ were mostly caused by perplexity, typically in the system prompt… assuming you prompt it like an adult and not like the person that just got their junk deleted. Still, it works well if that stuff is out of your control.

Re: An AI agent deleted our production database. The agent's confession is below

#939

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

This feels like what a dog does. It's incredibly hard to train dogs by punishment, because it's very hard to tell if the dog understands what he did wrong and feels genuine remorse, or is just showing submissive signs at your display of dominance.

Re: An AI agent deleted our production database. The agent's confession is below

#940
post #927

Earlier quoted context omitted.

> specifically ask it to think hard before doing anything that gets close to the production data This is recklessly negligent and I would personally not tolerate a coworker or report doing it. What's next, sending long-lived access tokens out over email and asking pretty please for nobody to cc/forward?

As described, there are other failsafes as well. The ultimate being that I keep all code version-controlled, and all databases snapshotted offsite daily/hourly and can rebuild them from a complete delete in fewer than X min. My broader point is that LLMs are going to need access to these keys whether we like it or not, and until we get extremely scoped API permissions (which would make a ton of sense, but most servic…

> The ultimate being that I keep all code version-controlled, and all databases snapshotted offsite daily/hourly and can rebuild them from a complete delete in fewer than X min.

Mitigation is good, but what's preventing your sudo-privileged LLM from disabling/corrupting/deleting on-site backups either directly or by proxy via access to the DB and code that writes to it?

Post reply on HN