Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

521–530 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#521

Post by fluoridation »

rowanG077 wrote: post #272
grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

Humans also don't follow given rules. Or we wouldn't need jail. We wouldn't need any security. We wouldn't need even user accounts.

Humans are able to follow rules. If you tell someone "don't press the History Eraser Button", and they decide they agree with the rule, they won't press the button unless by accident. If they really believe in the importance of the rule, they will take measures to stop themselves from accidentally press it, and if they really believe in the importance, they'll take measures to stop anyone from pressing it at all.

No matter how you insist to an LLM not to press the History Eraser Button, the mere fact that it's been mentioned raises the probability that it will press it.

Re: An AI agent deleted our production database. The agent's confession is below

#522

Post by nozzlegear »

anon84873628 wrote: post #482
coldtea wrote: post #449

Earlier quoted context omitted.

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

They should at least stop responding in the first person.

That's one of the first instructions in my system prompt when I'm working with an LLM:

> Do not reply in the first person – i.e. do not use the words "I," "Me," "We," and so on – unless you've been asked a direct question about your actions or responses.

It's not bulletproof but it works reasonably well.

Re: An AI agent deleted our production database. The agent's confession is below

#523

Post by r_lee »

smrtinsert wrote: post #503
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> "NEVER FUCKING GUESS" It's very hard to treat this post seriously. I can't imagine what harness if any they attempted to place on the agent beyond some vibes. This is "most fast and absolutely destroy things" level thinking. That the poster asks for journalists to reach out makes it like a no news is bad news publicity grab. Just gross. The AI era is turning about to be most disappointing era for software engineeri…

> The AI era is turning about to be most disappointing era for software engineering.

this has been obvious to me since like 2024, it truly is the worst, most uninspiring era of all time.

Re: An AI agent deleted our production database. The agent's confession is below

#524

Post by smrtinsert »

coldtea wrote: post #449
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

> The problem is millions of years of evolutionary wiring makes us see it as alive

Maybe for laymen, but I would think most technologists should understand that we're working with the output of what is effectively a massive spreadsheet which is creating a prediction.

Re: An AI agent deleted our production database. The agent's confession is below

#525

Post by zthrowaway »

dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

True but there’s nothing stopping a webdev dropping an API key in some wiki somewhere in the corporate intranet and the agent quickly picking that up.

Can you scan for that? Sure. But it’s a race to see who wins, the scanner or agent.

Re: An AI agent deleted our production database. The agent's confession is below

#526

Post by r-w »

simonjgreen wrote: post #423
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

Not a single mention of “maybe WE should have tested our backup strategy and scrutinised it”. Or even “maybe we should have backups away from the primary vendor”. Because this also says negligible DR and BC strategy. Complete accountability drop

  DROP TABLE Accountability;

Re: An AI agent deleted our production database. The agent's confession is below

#527

Post by hunterpayne »

throwdbaaway wrote: post #431
rahoulb wrote: post #12

Earlier quoted context omitted.

If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too. The LLM broke the safety rules it had been given (…

If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume. 1st hint - the API call only contains one volume: curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' 2nd hint - this gem from the tweet:…

"If I understand correctly, "

You don't. You are missing the part where the LLM had a token which blocked access as expected. Then the LLM searched the source base, found a different token with the delete privs and then used that.

PS That warning happens in staging envs too, the LLM doesn't know which env is which by design.

Re: An AI agent deleted our production database. The agent's confession is below

#528

Post by hn_c »

woeirua wrote: post #334

I love how the author took zero responsibility for anything that happened. Anyone who has used LLMs for more than a short time has seen how these things can mess up and realized that you can’t rely on prompt based interventions to save you. Guardrails need to be based on deterministic logic: - using regexes, - preventing certain tool or system calls entirely using hooks, - RBAC permission boundaries that prohibit age…

[dead]

Re: An AI agent deleted our production database. The agent's confession is below

#529

Post by ignoramous »

xmodem wrote: post #273
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Right. This line [0] from TFA tells me that the author needs to thoroughly recalibrate their mental model about "Agents" and the statistical nature of the underlying models.

[0] "This is the agent on the record, in writing."

Re: An AI agent deleted our production database. The agent's confession is below

#530

Post by hunterpayne »

threecheese wrote: post #451
oskarkk wrote: post #158

Earlier quoted context omitted.

It sounds like the keys just don't have any scoping. From the post: > The Railway CLI token I created to add and remove custom domains had the same volumeDelete permission as a token created for any other purpose. Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community…

Per their docs they have both “account” tokens and role-based tokens; the former have wide latitude (and might be used for DNS or root-access type stuff), while the latter are intended to be used for maintenance and have strong security boundaries. OP gave access to the former type without realizing it. In most orgs, those would be behind some escalation control. Unless the token creator didn’t know what they were do…

"which tracks for a non-expert"

So all agents then...because if you are an expert at a specific system, using a LLM probably slows you down, not speeds you up.

PS The article seems to imply that the token the LLM was given was a role based token. It then found ANOTHER token and used that instead.

Post reply on HN