Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#681

Post by 6r17 »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that you can call ..." ; at which point this tool just kills the agent. That said - this solution cannot guarantee by itself that the command is not ran ; but i'd argue that people have be writing more complex policies for ages - however the current LLM-era tend to produce the most competent idiots.

Re: An AI agent deleted our production database. The agent's confession is below

#682

Post by h14h »

Been meaning to check out Railway for a while, but now feeling happy about dragging my feet.

As flashy as their DX seems to be, the fact that a sketchy single VPS node with a server, a SQLite instance, and a LiteStream hookup has a better recovery story really makes me not trust their platform.

Re: An AI agent deleted our production database. The agent's confession is below

#683

Post by anonymars »

kokada wrote: post #365
computerdork wrote: post #285

Earlier quoted context omitted.

I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions. if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t be…

There is a difference between making a mistake like this one and being humble (e.g., lessons learned, having a daily external backup of the database somewhere else, or maybe asking the agent to not run commands directly in production but write a script to be reviewed later, or anything similar) and just blaming the AI and the service provider and never admitting your mistake like this article is all about. The fact t…

Indeed. I swear reality gets stranger and more implausible by the day.

"That isn't backups. That's a snapshot stored in the same place as the original — which provides resilience against zero failure modes that actually matter (volume corruption, accidental deletion, malicious action, infrastructure failure, the exact scenario we just lived through)."

Re: An AI agent deleted our production database. The agent's confession is below

#686

Post by jofzar »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

IMO the fail here is not having a true soft delete policy with a delete endpoint available

You need to protect customers from themselves. If you offer a true deletion endpoint/service you need to offer them a way to stop them from being absolute idiots when they inevitably cause a sev 0 for themselves.

Re: An AI agent deleted our production database. The agent's confession is below

#687

Post by refurb »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> If AI is physically capable of misbehaving, it might ($$1)

This is why all the “AI Armageddon” talk seems to silly to me.

AI is only as destructive as the access you give it. Don’t give it access where it can harm and no harm will occur.

Re: An AI agent deleted our production database. The agent's confession is below

#688

Post by niyikiza »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I have to agree here...of all things that went wrong here, I don't think the API surface is to blame. You need to have deterministic control & escalation mechanism on your agents whether they are calling an API or any other tool

Re: An AI agent deleted our production database. The agent's confession is below

#689

Post by cwsx »

noncoml wrote: post #295

"NEVER FUCKING GUESS!" "NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them." I can't help but laugh reading this. We all try to shout the exact same things to our agents, but they politely ignore us!

I also have to point out... "NEVER run destructive/irreversible *git* commands". So technically it DID follow the rules.

Re: An AI agent deleted our production database. The agent's confession is below

#690

Post by charcircuit »

Terr_ wrote: post #472
gigatree wrote: post #333

Earlier quoted context omitted.

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…

>it's just generating a plausible add-on to the document

A plausible document that follows the alignment that was done during the training process along with all of the other training where a LLM understanding its actions allows it to perform better on other tasks that it trained on for post training.

Post reply on HN