Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

261–270 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#261

Post by 827a »

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den.

> The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim:

Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes. The agent will never produce any output which will help you invoke future agents more safely, because to get to this point it has likely already bulldozed over multiple guardrails from Anthropic, Cursor, and your own AGENTS.md files. It still did it, because $$1: If AI is physically capable of misbehaving, it might. Prompting and training only steers probabilities.

Re: An AI agent deleted our production database. The agent's confession is below

#262

Post by techblueberry »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

> so you should expect your ceiling to spontaneously disintegrate any day,

I mean, I do?

Re: An AI agent deleted our production database. The agent's confession is below

#263

Post by Ekaros »

vbezhenar wrote: post #132

These stories make me rethink my approach to infra. I would never run AI with prod access, but my manager definitely has a way to obtain prod tokens if he really wanted to. Or if AI agent on his behalf wanted do. He loves AI and nowadays 80% of his messages were clearly made by AI. Sometimes I wonder if he's replaced by AI. And I can't stop them. So probably need to double down on backups and immutability...

Design, build an configure your infra in such a way that even if you wanted to destroy it you could not in too fast order. At least the unrecoverable bits and those you can not easily rebuild or replace.

Probably considering yourself as primary expert of system as threat actor is reasonable and thus you should be prevented yourself from being able to do irreparable damage.

Re: An AI agent deleted our production database. The agent's confession is below

#264

Post by chrsw »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

Ceilings do fall on people. LLMs do delete production databases. Will these things always inevitably happen? No, but the moment it does happen to someone I doubt they will be thinking about probabilities or Murphy's law or whatever.

I guess the question is, since we know these things can happen, however unlikely, what mitigations should be in place that are commensurate with the harms that might result?

Re: An AI agent deleted our production database. The agent's confession is below

#265

Post by iJohnDoe »

I only spent a few seconds reading this. These are off-the-cuff comments.

The model used is the most important part of the story.

Why is Cursor being mentioned at all? Doesn’t seem fair to Cursor.

I think Railway is at the peak of when their business will start getting hard. They’ve had great fun building something cool and people are using it. Now comes the hard part when people are running production workloads. It’s no longer a “basement self-hosting” business. They’ve had stability issues lately. Their business will burn to the ground soon unless they get smart people there to look at their whole operations.

Re: An AI agent deleted our production database. The agent's confession is below

#266

Post by fetzu »

jeremyccrane wrote: post #235
powera wrote: post #35

Earlier quoted context omitted.

He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.

In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?

Isn’t the point of an API to have two computers talk to each other? As in “if I want safeguards for humans, it would be my responsability to put them BEFORE calling that API”?

Re: An AI agent deleted our production database. The agent's confession is below

#267

Post by xmodem »

stratos123 wrote: post #236
maxbond wrote: post #213

Earlier quoted context omitted.

> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producin…

I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to th…

So it's only wrong in a technical and pedantic sense. A better phrasing might have been along the lines of "There are many sequences of tokens that will destroy your production database that are within the set of possible outputs"

Re: An AI agent deleted our production database. The agent's confession is below

#268

Post by conradfr »

adverbly wrote: post #21

This has to be fake right? Using LLMs for production systems without a sandbox environment? Having a bulk volume destroy endpoint without an ENV check? Somehow blaming Cursor for any of this rather than either of the above?

I'm half-convinced it's parody.

Re: An AI agent deleted our production database. The agent's confession is below

#269

Post by maxbond »

stratos123 wrote: post #236
maxbond wrote: post #213

Earlier quoted context omitted.

> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producin…

I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to th…

There will be details like rounding errors that will make certain sequences unreachable in practice, but that shouldn't provide you any comfort unless you know your dangerous outputs fall into that space. But they absolutely don't; the sequences we're interested in - well structured tool calls that contain dangerous parameters but are otherwise indistinguishable from desirable tool calls - are actually pretty probable.

The probability that an ideal, continuous LLM would output a 0 for a particular token in it's distribution is itself 0. The probability that an LLM using real floating point math isn't terrifically higher than 0.

Re: An AI agent deleted our production database. The agent's confession is below

#270

Post by iainmerrick »

jeremyccrane wrote: post #227
dentemple wrote: post #87

Earlier quoted context omitted.

It's one way for the company to make its money back, I guess.

Naw, we just want people to know. We followed all Cursor rules, thought we had protected all API keys, and trusted the backups of a heavily used infrastructure company. Cautionary tale sharing with others.

It’s a good cautionary tale -- in hindsight the danger signs are clear, but it’s also clear why you thought it was OK and how third parties unfortunately let you down.

The “agent’s confession” is the least interesting and useful part of the whole saga. Nothing there helps to explain why the disaster happened or what kind of prompting might help avoid it.

The key mistake is accidentally giving the agent the API key, and the key letdown is the lack of capability scoping or backups in the service.

The main lessons I take are “don’t give LLMs the keys to prod” and “keep backups”. Oh, and “even if you think your setup is safe, double-check it!”

Post reply on HN