Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

721–730 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#721

Post by dabinat »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I agree that this is the author’s fault considerably more than it is Railway’s, however I have learned from experience that no matter how many “are you sure you want to do this” prompts you have, sometimes users delete stuff they didn’t intend to delete and it’s better to not delete immediately but put it in a queue for deletion in a few hours and offer a way to reverse it. Even if it’s 100% user error, the user is very happy they didn’t lose data and the cost of storing it for an extra 5 hours or so is tiny.

Re: An AI agent deleted our production database. The agent's confession is below

#722

Post by grey-area »

rowanG077 wrote: post #272
grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

Humans also don't follow given rules. Or we wouldn't need jail. We wouldn't need any security. We wouldn't need even user accounts.

I don’t mean that in a small way (ie sometimes they don’t follow rules), I mean it in the more important sense that they don’t have a sense of right or wrong and the instructions we give them are just more context, they are not hard constraints as most humans would see them.

This leads to endless frustration as people try to use text to constrain what LLMs generate, it’s fundamentally not going to work because of how they function.

Re: An AI agent deleted our production database. The agent's confession is below

#723

Post by qsera »

coldtea wrote: post #550
smrtinsert wrote: post #524

Earlier quoted context omitted.

> The problem is millions of years of evolutionary wiring makes us see it as alive Maybe for laymen, but I would think most technologists should understand that we're working with the output of what is effectively a massive spreadsheet which is creating a prediction.

The thing with evolutionary wiring is that it doesn't matter if you're layman or "technologist". The technologist part is just a small layer on top of very thick caveman/animal insticts and programming. That's why a technologist can, just as easily as any layman, get addicted to gambling, or do crazy behaviors when attracted by the opposite sex.

>small layer on top of very thick caveman/animal insticts and programming.

Which is also why marketing and advertising works on EVERYONE. When AI puts out the phrase "Prompt engineering", everyone instinctively treat it as something deterministic, despite them having some idea of how an LLM works...

Re: An AI agent deleted our production database. The agent's confession is below

#724

Post by ArchOversight »

matheusmoreira wrote: post #649
compass_copium wrote: post #627

Earlier quoted context omitted.

I do not use claude and will use agents only when I am forced to, so I'm genuinely asking here: Can claude or other models not be run as a user or program with limited permissions? Do people just not bother to set it up? Why on earth would anyone run an RNG that can access $HOME/.ssh?

They absolutely can. I used to run Claude Code inside a firejail. Then I got paranoid to the point I developed my own virtual machine orchestration system just so I could run fully virtualized and isolated per-project Claude Code instances.

Do you have more information on this?

Re: An AI agent deleted our production database. The agent's confession is below

#725

Post by ImPostingOnHN »

xp84 wrote: post #708
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Verify? They should have attempted to drop the prod db with each token that they expected/hoped didn't have that permission?

If they wanted scoped tokens, they should have put on their roadmap an item to move to a SaaS product which has scoped tokens. Or ACLs. And until then, kept it on a list of risks: unscoped token may be misused by developer to delete prod db.

There's no difference in risk between this being done by an LLM vs. a human. Both make mistakes, so if you want to reduce the risk of this happening, you should poka-yoke[0] your systems to make this less likely to happen.

I'm not sure what's more striking about this blog post: that it includes virtually no assumption of blame on the part of the author, or that the author had this happen to them and was so angry with AI that they decided to use AI to write up the post.

0 – https://en.wikipedia.org/wiki/Poka-yoke

Re: An AI agent deleted our production database. The agent's confession is below

#726

Post by grey-area »

DiogenesKynikos wrote: post #587
getpokedagain wrote: post #341

Earlier quoted context omitted.

We are anthropomorphizing whenever we refer to prompts as instructions to models. They predict text not obey our orders.

> They predict text not obey our orders. Those are the same thing in this case. The latter is just an extremely reductionist description of the mechanics behind the former.

They are not in fact the same thing, and the difference is important.

They are certainly marketed as if they think, learn and follow orders, but they do not.

Re: An AI agent deleted our production database. The agent's confession is below

#727

Post by mtrifonov »

pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

Yes, you're right, in that there's no decision module separate from the output. It overcommits in the other direction.

The post-hoc reasoning the model produces when you ask "why did you do that" is also just text, and yet that text often matches independent third-party analysis of the same behavior at well above chance. If it really were uncorrelated text-completion, the post-hoc explanation should not align with the actual causes more than randomly. It does, frequently enough that I've stopped using it as evidence the user is naive.

"just outputs text" is doing more work than we acknowledge. The person asking the agent "why did you do that" might be an idiot for expecting anything more than a post-hoc rationalization, but that's exactly what you'd expect from a human too.

Re: An AI agent deleted our production database. The agent's confession is below

#728

Post by [deleted] »

SoftTalker wrote: post #648
prng2021 wrote: post #623

Earlier quoted context omitted.

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

For every API you publish, do you verify that scoped API keys work as they should before you go live? If so, why would you not do the same for APIs you integrate with? It's all part of "your" system from the user's perspective.

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#729

Post by mdavid626 »

lelanthran wrote: post #256
mdavid626 wrote: post #70

Earlier quoted context omitted.

In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation. That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing. The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"

The first delete would fail: “bucket not empty”. This might make the agent question the deletion (“bucket should be empty”).

Re: An AI agent deleted our production database. The agent's confession is below

#730

Post by anon84873628 »

prng2021 wrote: post #623
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

No I'm not. But it's clearly stated in the article that the API doesn't have scopes at all... So there was no reason to assume that some would be magically applied!

In GitHub or AWS etc you expect scopes to work because you define them. However if there is no way to define them in the first place, would you assume the system can somehow read your mind about what the client can access??

In fact I now believe this is a deliberate rhetorical sleight of hand. Point out a legit critique of the API design as if it is an excuse. But really any responsible engineer would notice the lack of scopes immediately, and that would be a flashing siren not to trust them to an agent.

Post reply on HN