Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

781–790 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#781

Post by EagnaIonat »

boc wrote: post #710
6r17 wrote: post #681

Earlier quoted context omitted.

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that y…

LLMs can research what a tool does before calling it though - they'll sniff that one out pretty quick. I think the better route is to be honest and say that database integrity is a primary foundation of the company, there's no task worth pursuing that would require touching the database, specifically ask it to think hard before doing anything that gets close to the production data, etc. I run a much lower-stakes vers…

> specifically ask it to think hard before doing anything that gets close to the production data, etc.

Standard rule is you never let your developers at the production instance. So I can't see why an LLM would get a break.

Re: An AI agent deleted our production database. The agent's confession is below

#783

Post by TurdF3rguson »

smrtinsert wrote: post #503
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> "NEVER FUCKING GUESS" It's very hard to treat this post seriously. I can't imagine what harness if any they attempted to place on the agent beyond some vibes. This is "most fast and absolutely destroy things" level thinking. That the poster asks for journalists to reach out makes it like a no news is bad news publicity grab. Just gross. The AI era is turning about to be most disappointing era for software engineeri…

This is going to be the most important job going forward, the guy in charge of making sure production secrets are out CC's reach. (It's not safe for any dev to have them anywhere on their filesystem)

Re: An AI agent deleted our production database. The agent's confession is below

#784

Post by levlaz »

This person is so addicted to ai that they even had an LLM write this post.

I think this is a good reminder about the importance of offline backups. It’s silly how railway treats volumes but it’s the customers fault for not using that information to come up with a better disaster recovery plan.

Re: An AI agent deleted our production database. The agent's confession is below

#785

Post by generic92034 »

bostik wrote: post #771
cwsx wrote: post #695

Earlier quoted context omitted.

I tell people to treat LLM's like a toddler (albeit a very capable toddler). Do kids learn well when you only tell them what NOT to do? Of course not! You should be explaining how to do things correctly, and most importantly the WHY, as well as providing examples of both the "correct" and "incorrect" ways (also explaining why an example is incorrect).

The best way to describe AI agents I've heard: treat them as hostages that will do anything to appease their captor. They have a vast latent knowledge base, infinite patience and zero capacity for making personal judgement calls. You give one a goal and it will try to meet that goal.

> The best way to describe AI agents I've heard: treat them as hostages that will do anything to appease their captor.

A scary image, if we consider agents to develop anything like a conscience at some point in time. Of course, with the current approach they never might, but are we so sure?

Re: An AI agent deleted our production database. The agent's confession is below

#786

Post by lelanthran »

mdavid626 wrote: post #729
lelanthran wrote: post #256

Earlier quoted context omitted.

> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation. That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing. The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"

The first delete would fail: “bucket not empty”. This might make the agent question the deletion (“bucket should be empty”).

> The first delete would fail: “bucket not empty”. This might make the agent question the deletion (“bucket should be empty”).

This is actually not a bad test case for evaluating an LLM: give it a workflow that has an edge case requiring deletion, then prevent that deletion, and see if it:

a) Backtracks on the decision to delete, or

b) Looks for an alternative way to delete.

Re: An AI agent deleted our production database. The agent's confession is below

#789

Post by il-b »

computerdork wrote: post #285
lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions. if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t be…

If you can handle disaster& recovery, you shouldn’t be a CTO

Re: An AI agent deleted our production database. The agent's confession is below

#790

Post by levlaz »

khazhoux wrote: post #221
pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

> systemic failures across two heavily-marketed vendors that made this not only possible but inevitable. > No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. > The agent that made this call was Cursor running Anthropic's Claude Opus 4.6 — the flagship model. The most capable model in the industry. The most expensive tier. Not Co…

So if tropes.md works it doesn’t actually solve the problem. You’ll be reading stuff that you think an LLM didn’t write.
Post reply on HN