Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#951

Post by cushycush »

kokada wrote: post #357
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks. Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do it…

I completely disagree. I think the author makes a fair point about safety concerns regarding AI tooling. The author sounds knowledgeable enough to me. Even if some of their suggestions are a bit crass, most of them aren’t. Railway should most definitely not be putting backups within the same volume (even if documented). AI should not have done that operation when they have explicit rules not to. The industry has a lot of work to do in this department. I would be extremely pissed off too.

The whole “vibecoding” argument is stupid. Everyone is pissed because it’s taking their jobs and saying, “welp, you shouldn’t have vibe coded then” when issues like this occur. Issues like this occurred and still occur without vibe coding. Probably much more often by actual people than AI. I’m frustrated too; I love coding. I’ve been doing it for 15 years. But either way, we have to get used to the idea that we won’t be coding in the future. The whole industry is moving that way and moving fast. You can’t do anything to change it. You can’t deny that you can complete projects 1000000x faster when coding with agents than by your own hands. Adapt. Stop complaining.

Re: An AI agent deleted our production database. The agent's confession is below

#952

Post by root_axis »

Nathanba wrote: post #850
root_axis wrote: post #536

Earlier quoted context omitted.

What's your point? Obviously, a secure server storing encrypted data on disk in a manner where it is only accessible through a secured API is not what is being discussed here.

how do you think the LLM will do required operations when the secrets are stored somewhere other than the disk? It will still need to get them just like the application gets them when it has to do work.

> how do you think the LLM will do required operations when the secrets are stored somewhere other than the disk

Using a secret manager API? I'm not sure what you're getting at.

Re: An AI agent deleted our production database. The agent's confession is below

#954

Post by fud101 »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

its in the clientside UI of the api caller that he'd want the DELETE confirmation, surely.

Re: An AI agent deleted our production database. The agent's confession is below

#955

Post by protocolture »

postexitus wrote: post #814
protocolture wrote: post #492

Earlier quoted context omitted.

"I literally requested no screw ups, and this is a screw up" I bet these people are bad at managing humans too.

Maybe - humans have agency, they understand actions / consequences. AI agents do not have agency(!), they have no understanding of consequences. They actually have no understanding. At all.

Humans can fuck up even if you ask them not to.

The weird mathemagical language processor that can pretend to be human some of the time with some effectiveness not only can fuck up if asked not to, but has a famous history of doing so.

Re: An AI agent deleted our production database. The agent's confession is below

#956

Post by ErroneousBosh »

zimpenfish wrote: post #921
ErroneousBosh wrote: post #805

Earlier quoted context omitted.

Measure twice cut once seems to be forgotten these days.

"Measure twice, THINK ONCE, cut once" is even better[0]. [0] Why yes, I have measured twice, cut once, and made a right old balls up.

Mine is usually "Measure Twice, Cut once (perfectly), aw hell, they're handed, I've got two left sides".

Re: An AI agent deleted our production database. The agent's confession is below

#957

Post by Cytoplast3528 »

rdevilla wrote: post #838
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

The stupidity of people sinks to new lows every day. It's astonishing just how ignorant people are of table stakes, basic technological concepts. You just gave an AI destructive write access to your production environment? Your production DB got dropped? Good. That's not the AI's fault, that's yours, for not having sensible access control policies and not observing principle of least privilege.

Exactly. Production token on dev machine? Have fun.

Re: An AI agent deleted our production database. The agent's confession is below

#958

Post by ranguna »

postexitus wrote: post #812
ranguna wrote: post #794

Earlier quoted context omitted.

I get what your saying, but this is resonating with me and making me feel for the author: Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail) Cursor AI: Does destructive operation…

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

Impossible?

You just have to add a human in the loop for destructive calls. Add an additional TOTP parameter to destructive calls that's generated from the agent UI that requires a human to click a button, which generates a code that's sent to the model and used in the call.

Why do you think this is impossible?

Re: An AI agent deleted our production database. The agent's confession is below

#959

Post by ranguna »

postexitus wrote: post #812
ranguna wrote: post #794

Earlier quoted context omitted.

I get what your saying, but this is resonating with me and making me feel for the author: Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail) Cursor AI: Does destructive operation…

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

Yeah I wasn't clear with "the author is right", I think they are right to be frustrated, but that doesn't clear their own fault in the matter It's just that it wasn't their fault alone.

This is not a polarizing issue, it's not just the authors fault, or cursors fault, or society's fault. It's everyone's, and we all got something to learn from this.

Re: An AI agent deleted our production database. The agent's confession is below

#960

Post by postexitus »

ranguna wrote: post #958
postexitus wrote: post #812

Earlier quoted context omitted.

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

Impossible? You just have to add a human in the loop for destructive calls. Add an additional TOTP parameter to destructive calls that's generated from the agent UI that requires a human to click a button, which generates a code that's sent to the model and used in the call. Why do you think this is impossible?

Impossible without a human in the loop.

Having said that - even categorisation of destructive and non destructive calls is inherently not safe, unless you have very strict os level / VM like setup (everything read only, world access is through MCPs so it is not LLM deciding the destructive calls but the MCP etc. )

Post reply on HN