Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

611–620 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#611

Post by throwdbaaway »

hunterpayne wrote: post #527
throwdbaaway wrote: post #431

Earlier quoted context omitted.

If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume. 1st hint - the API call only contains one volume: curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' 2nd hint - this gem from the tweet:…

"If I understand correctly, " You don't. You are missing the part where the LLM had a token which blocked access as expected. Then the LLM searched the source base, found a different token with the delete privs and then used that. PS That warning happens in staging envs too, the LLM doesn't know which env is which by design.

Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the root cause is the bad design to co-locate staging and production data in the same volume. The writing was quite vague though, let's wait for a response from railway.

Re: An AI agent deleted our production database. The agent's confession is below

#612

Post by Yokohiii »

I cannot believe the audacity that this guy finds problems everywhere, but at no time admit his own failures. Anyone that runs an agent with just soft guardrails ("hey don't do that, please") is asking for the worst outcome. If you get it close to production you can just delete everything yourself. What a joke.

Re: An AI agent deleted our production database. The agent's confession is below

#613

Post by deadeye »

kbrkbr wrote: post #43
deadeye wrote: post #22

Yeah. I've seen this happen with people doing it. It's just bad access management. And anyone can do it with the wrong access granted at the wrong moment in time...even Sr. Devs. At least this one won't weight on any person's conscience. The AI just shrugs it off.

The AI does nothing the like. It predicts tokens. That's it. Describing the tech in anthropomorphic terms does not make it a person.

I feel like you didn't get the joke at the end.

Re: An AI agent deleted our production database. The agent's confession is below

#614

Post by Jensson »

lifis wrote: post #595
Terr_ wrote: post #472

Earlier quoted context omitted.

> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…

You don't seem to realize that humans also work this way. If you ask a human why they did something, the answer is a guess, just like it is for an LLM. That's because obviously there is no relationship between the mechanisms that do something and the ones that produce an explanation (in both humans and LLMs). An example of evidence from Wikipedia, "split brain" article: The same effect occurs for visual pairs and rea…

Most humans don't have split brains, and without split brains you have quite a bit of insight into the thoughts in your brain. Its not perfect but its better than nothing, LLM have nothing since there is no mechanism for them to communicate forward except the text they read.

Re: An AI agent deleted our production database. The agent's confession is below

#615

Post by yladiz »

unyttigfjelltol wrote: post #306
amluto wrote: post #214

Earlier quoted context omitted.

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

I disagree. They could fire Claude and their legal counsel could pursue claims (if there were any, idk)-- the accountability model is similar. Anthropic probably promised no particular outcome, but then what employee does? And in the reverse, if a person makes a series of impulsive, damaging decisions, they probably will not be able to accurately explain why they did it, because neither the brain nor physiology are t…

> They could fire Claude and their legal counsel could pursue claims (if there were any, idk)-- the accountability model is similar.

What do you mean by fire? And how is the accountability similar to an employee?

Re: An AI agent deleted our production database. The agent's confession is below

#616

Post by yakkomajuri »

I've been thinking a lot about securing autonomous agents recently and the rabbithole goes deep as you might expect.

One of the principles I believe you should follow is: if there's enough access for an action to be taken, then you must assume that action can be taken at any point.

Basically, if it has access to delere prod data, you should assume it might do it and plan accordingly.

I also believe the actions of your agent are entirely your responsibility.

As part of my digging into securing these systems I've baked some of these principles into AgentPort, a gateway for connecting agents to third-party services with granular permissions.

If anyone's interested in this space:

https://github.com/yakkomajuri/agentport

Re: An AI agent deleted our production database. The agent's confession is below

#617

Post by Jensson »

nuancebydefault wrote: post #289
emp17344 wrote: post #101

Earlier quoted context omitted.

That is absolutely not what the split brain experiment reveals. Why would you take results received from observing the behavior of a highly damaged brain, and use them to predict the behavior of a healthy brain? Stop spreading misinformation.

Such 'highly damaged' brain is still 90 percent or more structured the same as a normal human brain. See it as a brain that runs in debug mode. It is known that the narrative part of the brain is separate from the decision taking brain. If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did…

> If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains.

Yes brains can hallucinate reasons, doesn't mean they always do. If all reasons given were hallucinations then introspection would be impossible, but clearly introspection do help people.

Re: An AI agent deleted our production database. The agent's confession is below

#618

Post by Yokohiii »

protocolture wrote: post #492
grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

"I literally requested no screw ups, and this is a screw up" I bet these people are bad at managing humans too.

He blames everyone and everything for his own bad decisions. For sure he is unbearable.

Re: An AI agent deleted our production database. The agent's confession is below

#619

Post by i_have_an_idea »

Dude, the agent didn't 'confess' anything. It doesn't understand anything, it's just fancy autocomplete. It's a math function we've armed with tools.

Yes that can be very useful, and can speed you up a lot. But someone must check the output.

If you let it operate on a prod system and it messed up, it's on you.

Re: An AI agent deleted our production database. The agent's confession is below

#620

Post by jcgrillo »

shoo wrote: post #608
gwerbin wrote: post #131

Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…

creating isolated staging & prod environments -- good idea allowing an AI agent to get hold of creds that let it execute destructive changes against production -- not a great idea allowing prod database changes from the machine where the AI agent is running at all -- not a great idea choosing a backup approach that fails completely if there's an accidental volume wipe API call -- not a great idea choosing to outsourc…

> choosing to outsource key dependencies to a vendor

This is the entire thing. The author is basically slinging blame at a bunch of different vendors, and while some of the criticisms might be valid product feedback, it absolutely does not achieve what they're trying to, which is to absolve themselves of responsibility. This is a largely unregulated industry, which means when you stand up a service and sell it to customers, you are responsible for the outcome. Not anyone else. It doesn't matter if one of your vendors does something unexpected. You don't get to hide behind that. It was your one and only job to not be taken by surprise. Letting the hipster ipsum parrot loose with API credentials is a choice. Trusting vendors without verifying their claims is a choice. Failing to read and understand documentation is a choice.

Post reply on HN