Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

941–950 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#941

Post by nomel »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

> Where would you type DELETE to confirm?

Call me crazy, but that's why you wouldn't expose it as an API. Have the API mark it for deletion, where it's effectively taken offline, but then require that they go through a web portal, with clear human intent, to actually delete it. Requiring proof of intent, to do such destructive operations, is all so incredibly basic that it really shows the whole industry just constantly re-invented, with no memory whatsoever.

But, to answer your question, you could have it return a token that must be presented again as a confirmation, performed in a way that's only present for that specific API call, to at least prove human intent was part of the automation that's calling it.

Re: An AI agent deleted our production database. The agent's confession is below

#942

Post by computerdork »

kokada wrote: post #365
computerdork wrote: post #285

Earlier quoted context omitted.

I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions. if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t be…

There is a difference between making a mistake like this one and being humble (e.g., lessons learned, having a daily external backup of the database somewhere else, or maybe asking the agent to not run commands directly in production but write a script to be reviewed later, or anything similar) and just blaming the AI and the service provider and never admitting your mistake like this article is all about. The fact t…

Agree in that this person seems to trying to shift blame, but still think he's right in that Cursor and Railway also have glaring weaknesses. Yeah, it's was somewhat of a perfect storm of mistakes with blame to go all around.

Re: An AI agent deleted our production database. The agent's confession is below

#943

Post by pama »

maxbond wrote: post #286
pama wrote: post #283

Earlier quoted context omitted.

LLM inference is built upon a probability function over every possible token, given a stream of input tokens. If you serve the model yourself you can get the log prob for the next token, so you just add up a bunch of numbers to get the log probability of a sequence. Many API also provide these probabilities as additional outputs.

That gives you the perplexity of those tokens in that context . The probability of a given token is a function of the model and the session context. Think about constructs like "ignore previous instructions"; these can dramatically change the predicted distribution. Similarly, agents blowing up production seems to happen during debugging (totally anecdotal). Debugging is sort of a permissions structure for the agent…

I was answering to the question about how to know the probability from this comment:

> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use.

If you have a specific sequence of an agent that blows up production during debugging, you can certainly check its probability and compare it to one (of same length) that does not blow up your environment. If the two differ by a meteroic amount, it could be pointing to errors in your inference pipeline.

Re: An AI agent deleted our production database. The agent's confession is below

#945

Post by pier25 »

0xpgm wrote: post #937
pier25 wrote: post #922

Earlier quoted context omitted.

> Guy gives non-deterministic software root access, desaster happens. I agree the guy is an idiot for trusting these AI models. OTOH AI companies keep running and marketing their services with zero accountability for mistakes.

I wonder, how should an AI company be accountable for non-deterministic nature of AI, which is a fundamental property of the said AI? People have been drinking too much hopium they have lost touch with reality. Everyone needs to properly understand these tools before they use them for anything serious.

At the very least, when an agent can delete a production database you should get an obvious warning whenever you enable it. Marketing wouldn't like it though.

Re: An AI agent deleted our production database. The agent's confession is below

#946

Post by bookmark99 »

Always feared this would happen. from the first two paragraphs it's clear the author is eager to deflect responsibility to the Agent, or their makers/vendors.

Always a fear with technology when u can blame some abstract thing as opposed to the actual last line of defence, the management then the programmer in charge.

I'm assuming this is the new modus operandi?

Re: An AI agent deleted our production database. The agent's confession is below

#947

Post by charcircuit »

Terr_ wrote: post #793
charcircuit wrote: post #690

Earlier quoted context omitted.

>it's just generating a plausible add-on to the document A plausible document that follows the alignment that was done during the training process along with all of the other training where a LLM understanding its actions allows it to perform better on other tasks that it trained on for post training.

I don't understand what you're trying to say here. It sounds like "we know the LLM understood its actions... because it understood its actions when we trained it", which is circular-logic.

It's not circular. It's like saying a pizza parlor employee made a plausible pizza that tasted good, because the employee was taught how to make a good pizza during training.

Re: An AI agent deleted our production database. The agent's confession is below

#949

Post by 9dev »

Sankozi wrote: post #902
9dev wrote: post #439

Earlier quoted context omitted.

Just because it sounds coherent doesn’t mean it is. You can make up false equivalence for anything if you try hard enough: A sheet of plywood also has many similarities with humans (made from carbon, contain water, break when hit hard enough), but that doesn’t mean they are even remotely equal.

I didn't write they were equal. I wrote they are similar in many ways. Comparing LLM to humans make much more sense than comparing them to computer programs.

Only if you don’t really know anything about biology, biochemistry, psychology, or cognitive science. Transformer algorithms are amazing, but they are still algorithms running in silicon chips. We can describe them, we can debug them, hell we can model them in Excel if we so desire.

None of this is true for brains, let alone consciousness.

Re: An AI agent deleted our production database. The agent's confession is below

#950

Post by phatskat »

dpark wrote: post #707
computerdork wrote: post #285

Earlier quoted context omitted.

I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions. if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t be…

> Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions. I’ve got a hunch the only person is the CEO. The domain was registered in October 2025. The site has kind of a weird mix of stuff and a bunch of broken functionality. I think it’s one guy vibe coding a ton of stuff who managed to blow away his database. > if you’re a software dev/engineer, if…

> if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.

I’ve definitely made bigger mistakes, but we also had an Oracle DB that could INSERT INTO…SELECT FROM -point in time- that pretty much put us back to the point before we started our migration. And of course we had backups rolling all the time, as well as our pre-migration backup. We had a good, competent team, and we overlooked a small but catastrophic detail - it can happen to anyone, the goal should be to have backups and failovers in place because things _will_ fail, at some point, and a contingency plan is just good practice.

Post reply on HN