Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

911–920 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#911

Post by CivBase »

rowanG077 wrote: post #272
grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

Humans also don't follow given rules. Or we wouldn't need jail. We wouldn't need any security. We wouldn't need even user accounts.

Humans understand rules to be commands with risks and consequences. They conceously evaluate the benefits of breaking rules against the risks and consequences. They also have their own needs, self-interests, and instincts for preservation and community.

LLMs don't do or have any of this. To them "rules" (just like all prompts) are just weights on a graph traversal used to output text.

They are not the same.

Re: An AI agent deleted our production database. The agent's confession is below

#912

Post by lesostep »

margalabargala wrote: post #377
nkrisc wrote: post #323

Earlier quoted context omitted.

> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics. Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material. Barring that, it i…

I have lived about 40 years beneath ceilings and never personally taken a preventative measure. I allow my kids to walk under not only our own ceiling, but other people's ceilings, and I have never asked those people if their ceilings were properly maintained.

It's a basic "When you invent the ship, you also invent the shipwreck" principle.

Ceilings are usually build to be predictably collapsible and not to cause much damage. You will hear the cracking and see the sagging long before it will collapse, that's why you are reasonably safe walking under ceiling that looks good. If you never taken a preemptive measure to not go under unstable looking roof, that's on you. Or maybe on people that track that kind of thing and repair before damage is done.

LLMs will delete you prod, if given permission, so we need same engineering principles applied there as well. We need warning signs that something will collapse soon. We need to know what relatively safe ceiling collapsing looks like.

We are not some people walking in homes with a ceiling. We supposed to be people that build this houses and repair them in time, so the ceiling wouldn't collapse on the user heads!

As of right now, LLMs are shitty ceilings and shouldn't be given any access to prod.

Re: An AI agent deleted our production database. The agent's confession is below

#913

Post by 8note »

hacker161 wrote: post #894
eloisius wrote: post #852

Earlier quoted context omitted.

I can’t say for sure, but I think Claude’s mode is nothing more than part of the system prompt. I don’t think it actually takes away web request or file write tools. I say this because I could swear I’ve seen Claude go ahead and make some changes even while we’re in plan mode. Web requests certainly, because it can fetch docs and so forth.

You’re not alone, I’ve absolutely seen the same behavior occasionally with Opus in OpenCode where it takes actions it shouldn’t be able to in plan mode.

that sounds like opencode has a privilege bug too?

Re: An AI agent deleted our production database. The agent's confession is below

#914

Post by rfwhyte »

What an utterly feckless and deflection filled response from the person writing this article.

Literally zero personal accountability for the choices they themselves made that led to this outcome.

"Jer" could have chosen to hire actual human developers who almost certainly wouldn't have deleted his production database, but instead, he chose to cut corner and use AI all so he could make himself more money, and when it finally came back to bite him in the ass it suddenly became everyone else's fault.

Re: An AI agent deleted our production database. The agent's confession is below

#915

Post by pxc »

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.

> The "system rules" the agent is referring to are consistent with Cursor's documented system-prompt language and our project rules for this codebase. Both safeguards failed simultaneously.

It seems like human brains aren't built for the experiences we get with AI agents, where "you can just tell them to do something, and they do it!"... until you can't. It's not a junior dev, it's demented. It's not a magical assistant, it's a demonic assistant, possessed by strange forces that act unexpectedly. All possible metaphors are bad.

I've been reading articles and listening to interviews by a prominent AI booster lately (Yegge), and he talks about a kind of curve of engagement with LLM agents in which "trust goes up", and you delegate more and more work to the LLM as you progress along this curve.

One of the things that always struck me (and struck me as wrong) about his characterization is that running agents in YOLO mode arrives super, super early. It's either the second step or implicit in the first "stage". Why don't people see external sandboxing (or, like the article suggests "auditing token scopes") as a prerequisite to running these agents in environments that have access to production (let alone YOLO modes)? How can the standard answer from AI boosters just be "you WILL lose data. it's a brave new world!"? It's possible to use them without being totally careless. Why not try that?

Re: An AI agent deleted our production database. The agent's confession is below

#916

Post by rfwhyte »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

They relied on frontier tech because GREED. Let's not kid ourselves that the decision to use AI here was done for any other reason than it would save this company the labor costs of actual human employees. They decided their profit was more important than the security of their customer's data, and now they are suffering the well deserved consequences for it.

Re: An AI agent deleted our production database. The agent's confession is below

#917

Post by 8note »

robertkarl wrote: post #59

PocketOS's website says "Service Disruption: We're currently experiencing a major outage caused by an infrastructure incident at one of our service providers. We are actively working with their team on recovery. Next update by 10:00a pst." This is wrong. It was not an infra incident at their service provider. As Jer says in the article, their own tooling initiated the outage. And now they're threatening to sue? "We'v…

itll be entertaining if someone points at this thread as "the operator has no idea what they are doing and followed 0 best practices for software engineering, and anti-patterns for agentic ai"

Re: An AI agent deleted our production database. The agent's confession is below

#918

Post by aeve890 »

juliansark wrote: post #903

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

>total and absolute failure of guard rails

It seems here the guard rails at failure were the llm users right? Whatever guard rails you can think may be useless against the superior human stupidity.

Also, what's the LLM use policy at the SD-6?

Re: An AI agent deleted our production database. The agent's confession is below

#919

Post by ericd »

kikimora wrote: post #867
ericd wrote: post #656

Earlier quoted context omitted.

Yes, it needs to be sandboxed very carefully. It should have no way to access anything outside of the directories you mount in the sandbox.

I wonder what is the approach you taking? In my dev env we have .env files that supposed to have dev api keys for staging and testing. Production parameters stored in parameter store. There is also deploy script, that can deploy into production given there is a token in AWS CLI. I understand there is a way to keep Claude inside working dir. but how to limit it from accidentally deploying production, modifying terrafo…

I only run claude code inside a docker container that only mounts the directory it's called in, and I make damn sure I don't run it in a way to mount a directory that has any creds in it other than dev infra. Do not mount a home directory with a bunch of . directories (.aws, .ssh, etc). The nice thing about the docker containers otherwise is you need to explicitly choose what to pass in, but getting lazy and passing in things just in case or because it's convenient is asking for trouble.

Re: An AI agent deleted our production database. The agent's confession is below

#920

Post by 8note »

hunterpayne wrote: post #516
zkmon wrote: post #414

The biggest rule-break was done, not by the agent or infra company, but by the person who gave such elevated authorization (API key) to an autonomous bot.

That's not what happened.

if an api key with full perms was put in a place where the agent can access it, that is the biggest problem.

that somebody made a key thst can delete prod when they dont need to delete prod is the underlying problem with that

and underlying that still is that the staging environments were on the same account as prod.

Post reply on HN