Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

661–670 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#661

Post by Kim_Bruning »

j-bos wrote: post #654
Kim_Bruning wrote: post #551

Earlier quoted context omitted.

I agree to the practical part of this, with two nuances: The full data of what's in an LLM's "consciousness" is the conversation context. Just because it isn't hidden, doesn't necessarily mean it doesn't contain information you've overlooked. Asking "why did you do that" won't reveal anything new, but it might surface some amount of relevant information (or it hallucinates, it depends which LLM you're using). "Analys…

> The full data of what's in an LLM's "consciousness" is the conversation context. No it's not, see research on hiddens states using SAE's and other methods. TBC, I agree with your second point, though I still believe top level OP was reckless and is now doing the businessman's version of throwing the dog under the bus.

We might actually be in full agreement. You can't get a faithful replay of these internal states. They're gone at end of generation. You can only query and re-derive from the visible context. Hence limited (though not zero) utility, depending on model, harness, and prompt.

Re: An AI agent deleted our production database. The agent's confession is below

#662

Post by tgsovlerkhgsel »

Blaming Railway for this feels a bit off... criticizing that they advertise the API for MCP use is valid, criticizing the lack of ability to set more granular permissions is valid - but complaining that an API call doesn't come with a confirmation prompt, or that after you deleted your data the infrastructure provider takes time to figure out whether they can use their backup to undo your mistake?

With a major provider, there would be a "recovery SLA", and it would be "we guarantee that once you make the delete call we won't be able to get your data back".

What I'm missing in this article is "we fucked up by not having actual, provider-independent, offline backups newer than 3 months". They'd have the same result if a rogue employee or ransomware actor got access to their Railway account, or Railway accidentally deleted their account, Railway went down, etc.

Re: An AI agent deleted our production database. The agent's confession is below

#665

Post by majormajor »

prng2021 wrote: post #623
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

I think the suggestion from their "somewhat sympathetic" position is that if you are integrating with something you should (a) find out up front what limits it does or doesn't have on its API keys, so that it's not a nasty surprise later, and (b) absolutely don't give keys without really tight scopes to "agents."

The person here who deleted prod DB with their agent made an assumption that an API key wouldn't have broad permission if there weren't warnings ("We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete. "). I don't know what the UI looks like exactly, but unless I'm explicitly selecting a specific set of limited permissions, I don't know why I'd assume "this won't do more than I am creating it for". Like "I didn't ask the guy at the gun store to put bullets in, I wouldn't have given the gun to the agent if I'd known there were bullets in it."

I also would be wary of running on an "infrastructure provider" that didn't make things like that very clear.

Is this overly harsh? I don't know. I've had to explain far too many times to people (including other engineers) what makes doing certain things unsafe/foolish (since they initially think I'm wasting time checking things like that). So I think stories like this need to be taken as "absolutely do not make the same mistakes" cautionary tales by as many people as possible.

Re: An AI agent deleted our production database. The agent's confession is below

#666

Post by dpark »

smj-edison wrote: post #547
crazygringo wrote: post #313

Earlier quoted context omitted.

Yup, this is bizarre. A top use case for needing a backup is when you accidentally delete the original. You need to be able to delete backups too, of course, but that absolutely needs to be a separate API call. There should never be any single API call that deletes both a volume and its backups simultaneously. Backups should be a first line of defense against user error as well. And I checked the docs -- they're call…

Plus backups should be time gated, where the software physically blocks you from removing backups for X days.

This is one of those things that seems like a good idea on the surface but is rife with problems.

Does the company hosting the backups do it for free? Or do they charge their customers to keep holding onto backups they no longer want?

Is “my DB company refuses to delete the data” a valid legal response to a copyright enforcement or a GDPR demand?

Re: An AI agent deleted our production database. The agent's confession is below

#669

Post by preperat »

The Railway detail is the part that sticks. Backups stored inside the same volume they're backing up isn't really a backup, it's a snapshot with extra steps. Delete the volume, delete the evidence. That said, credential scoping should have been the first line here. A token that can destroy production infrastructure shouldn't exist in a dev environment config, full stop.
Post reply on HN