Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

451–460 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#451

Post by threecheese »

oskarkk wrote: post #158
rahoulb wrote: post #12

Earlier quoted context omitted.

If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too. The LLM broke the safety rules it had been given (…

It sounds like the keys just don't have any scoping. From the post: > The Railway CLI token I created to add and remove custom domains had the same volumeDelete permission as a token created for any other purpose. Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community…

Per their docs they have both “account” tokens and role-based tokens; the former have wide latitude (and might be used for DNS or root-access type stuff), while the latter are intended to be used for maintenance and have strong security boundaries. OP gave access to the former type without realizing it.

In most orgs, those would be behind some escalation control. Unless the token creator didn’t know what they were doing/creating, which tracks for a non-expert.

Re: An AI agent deleted our production database. The agent's confession is below

#452

Post by drdaeman »

dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

> This is a modern “oops, I ran DROP TABLE on the production database” story.

It's not that story, though. It's a story "oops, my tool ran DROP TABLE on the production database" (blaming the tool). At least I haven't heard people blaming their terminals or database clients as if the tool is somehow responsible for it.

Re: An AI agent deleted our production database. The agent's confession is below

#455

Post by fluoridation »

narrator wrote: post #413
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

It's also important to realize that AI agents have no time preference. They could be reincarnated by alien archeologists a billion years from now and it would be the same as if a millisecond had passed. You, on the other hand, have to make payroll next week, and time is of the essence.

How is that relevant, though?

Re: An AI agent deleted our production database. The agent's confession is below

#456

Post by Zak »

pessimizer wrote: post #447
jumpconc wrote: post #303

Earlier quoted context omitted.

You haven't met certain humans. Not all humans have internal capacity for accountability. The real meaning of accountability is that you can fire one if you don't like how they work. Good news! You can fire an AI too.

Bad news! They will not be aware that you have done this and will not care.

The purpose of firing a person shouldn't be vengeance but to remove someone who is unreliable or not cost effective.

It's similarly reasonable to drop a tool that's unreliable, though I don't think that's a reasonable description here. Instead, they used a tool which is generally known to be unpredictable and failed to sandbox it adequately.

Re: An AI agent deleted our production database. The agent's confession is below

#457

Post by gizmondo »

jeremyccrane wrote: post #232
easton wrote: post #79

Earlier quoted context omitted.

AWS actually has a thingy on some services called “deletion protection” to prevent automation from accidentally wiping resources the user didn’t want it to (you set the bit, and then you need to make a separate api request to flip the bit back before continuing). I think it’s designed for things like Terraform or CloudFormation where you might not realize the state machine decided your database needed to be replaced…

This should be the solution. All destructive actions require human intervention.

A human representing the company should be physically present in the provider's office to perform such an action or what? Otherwise you would just grant your agent a way to impersonate a human.

Re: An AI agent deleted our production database. The agent's confession is below

#458

Post by operatingthetan »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> Lord, even calling it a "confession" is so cringe. The agent is not alive.

The AI companies are very invested in anthropomorphizing the agents. They named their company "Anthropic" ffs. I don't blame the writer for this, exactly.

Re: An AI agent deleted our production database. The agent's confession is below

#460

Post by josephg »

gbnwl wrote: post #444
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

The entire post reads like it was generated via LLM as well.

It clearly was, at least in part. Somehow, it feels just right here: Man trusts AI to do the right thing and it burns him. 5 minutes later, man trusts AI to explain what happened on X.

Its a greek tragedy in 2 acts.

Post reply on HN