Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

251–260 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#251

Post by reliablereason »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

Right! Blaming an agent or anyone else is crazy. The author built a system that had the capability of deleing the prod database.

The system did delete the database cause the author built it like that.

Re: An AI agent deleted our production database. The agent's confession is below

#252

Post by jeremyccrane »

rahoulb wrote: post #12
heliumtera wrote: post #7

Someone trusted prod database to an llm and db got deleted. This person should never be trusted with computers ever again for being illiterate

If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too. The LLM broke the safety rules it had been given (…

Bingo.

Re: An AI agent deleted our production database. The agent's confession is below

#254

Post by ses1984 »

caminante wrote: post #229
yongjik wrote: post #198

Earlier quoted context omitted.

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong." Actual quote: > “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”

Engineering controls basically mean making it impossible to do something in a way that results in catastrophe.

Re: An AI agent deleted our production database. The agent's confession is below

#256

Post by lelanthran »

mdavid626 wrote: post #70
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing.

The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"

Re: An AI agent deleted our production database. The agent's confession is below

#257

Post by anygivnthursday »

tee-es-gee wrote: post #199
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

I do think that as service providers we now have a new "attack vector" to be worried about. Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, because generally users won't do such a destructive action via the API or if they do, they likely understand the consequences. Or at the very least don't complain if they do it without reading the docs carefully enough. But n…

> Up to now, having an API that deletes the whole volume, including backups, might have been acceptable

It was never acceptable, major service providers figured this out long time ago and added all sorts of guardrails long before LLMs. Other providers will learn from their own mistakes, or not.

Re: An AI agent deleted our production database. The agent's confession is below

#258

Post by _pdp_ »

What do you expect?

We give a non-deterministic system API keys that 99.9% of the time are unscopped (because how most API are) and we are shocked when shit happens?

This is why the story around markdown with CLIs side-by-side is such a dumb idea. It just reverses decades of security progress. Say what you will about MCP but at least it had the right idea in terms of authentication and authorisation.

In fact, the SKILLS.md idea has been bothering me quite a bit as of late too. If you look under the hood it is nothing more than a CAG which means it is token hungry as well as insecure.

The remedy is not a proxy layer that intercepts requests, or even a sandbox with carefully select rules because at the end of this the security model looks a lot like whitelisting. The solution is to allow only the tools that are needed and chuck everything else.

Re: An AI agent deleted our production database. The agent's confession is below

#259

Post by oskarkk »

gwerbin wrote: post #131

Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…

> what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token

It sounds like the token the author created just didn't have any scope, it had full permissions. From the post:

> Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community has been asking for scoped tokens for years. It hasn't shipped.

So it wasn't "a narrowly scoped API token", it was a full access token, and I suspect the author didn't have any reason to think it was some special specific purpose token, he just didn't think about what the token can do. What he's describing is his intent of creating the token (how he wanted to use it), not some property of the token.

Author said in an X post[0] that it was an "API token", not a "project token", which allows "account level actions"[1], with a scope of "All your resources and workspaces" or "Single workspace"[2], with no possibility of specifying granular permissions. Account token "can perform any API action you're authorized to do across all your resources and workspaces". Workspace token "has access to all the workspace's resources".

[0] https://x.com/lifeof_jer/status/2047733995186847912

[1] https://docs.railway.com/cli#tokens

[2] https://docs.railway.com/integrations/api#choosing-a-token-t...

Re: An AI agent deleted our production database. The agent's confession is below

#260

Post by lelanthran »

jeremyccrane wrote: post #235
powera wrote: post #35

Earlier quoted context omitted.

He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.

In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?

> Why would an API key (regardless of its scope) be able to delete without confirmation?

How do you see this working? Any confirmation would be given by the agent.

Post reply on HN