Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

291–300 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#291

Post by TZubiri »

>Railway's failures (plural)

>This is not the first time Cursor's safety has failed catastrophically.

How can you lack so much self awareness and be so obtuse.

There's no section "Mistakes we've made" and "changes we need to make"

1. Using an llm so much that you run into these 0.001% failure modes. 2. Leaking an API key to an unauthroized LLM agent (Focus on the agent finding the key? Or on yourself for making that API key accessible to them? What am I saying, in all likelihood the LLM committed that API key to the repo lol) 3. Using an architecture that allows this to happen. Wtf is railway? Is it like a package of actually robust technologies but with a simple to use layer? So even that was too hard to use so you put a hat on a hat?

Matthew 7:3 “Why do you look at the speck of sawdust in your brother’s eye and pay no attention to the plank in your own eye?."

Re: An AI agent deleted our production database. The agent's confession is below

#292

Post by jbxntuehineoh »

tee-es-gee wrote: post #199
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

I do think that as service providers we now have a new "attack vector" to be worried about. Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, because generally users won't do such a destructive action via the API or if they do, they likely understand the consequences. Or at the very least don't complain if they do it without reading the docs carefully enough. But n…

it's a great source of schadenfreude though, I love watching vibecoders get their shit nuked

Re: An AI agent deleted our production database. The agent's confession is below

#294

Post by [deleted] »

keeda wrote: post #290
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#295

Post by noncoml »

"NEVER FUCKING GUESS!" "NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them."

I can't help but laugh reading this. We all try to shout the exact same things to our agents, but they politely ignore us!

Re: An AI agent deleted our production database. The agent's confession is below

#296

Post by gpm »

keeda wrote: post #290
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

An eager intern can remember things you tell beyond that which would fit in an hours conversation.

A disgruntled employee definitely remembers things beyond that.

These are a fundamentally different sort of interaction.

Re: An AI agent deleted our production database. The agent's confession is below

#297

Post by TZubiri »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

It's as if they internalized a post-mortem process that is designed to find root causes, but they use it to shift blame into others, and they literally let the agent be a sandbag for their frustrations.

THAT SAID, it does help to let the agent explain it so that the devs perspective cannot be dismissed as AI skepticism.

Re: An AI agent deleted our production database. The agent's confession is below

#298

Post by crazygringo »

The post overall is interesting, but this:

> A single API call deletes a production volume. There is no "type DELETE to confirm." There is no "this volume is in use by a service named [X], are you sure?" There is no rate-limit or destructive-operation cooldown.

...makes me question the author's technical competence.

Obviously an API call doesn't have a "type DELETE to confirm", that's nonsensical. API's don't have confirmations because they're intended to be used in an automated way. Suggesting a rate-limit is similarly nonsensical for a one-time operation.

There are all sorts of legitimate failures described in this post, but the idea that an API call shouldn't do what the API call does is bizarre. It's an API, not a user interface.

Re: An AI agent deleted our production database. The agent's confession is below

#300

Post by cortesoft »

iainmerrick wrote: post #274
cortesoft wrote: post #188

Earlier quoted context omitted.

> Whereas for an agent it will happily include details that are not literally in its chain of thought as justifications for its decisions. Humans do this too, ALL THE TIME. We rationalize decisions after we make them, and truly believe that is why we made the decision. We do it for all sorts of reasons, from protecting our ego to simply needing to fill in gaps in our memory. Honestly, I feel like asking an AI it’s tr…

Plan for a world where we are just as stupid tomorrow as we are today. In other words, the action item can’t be “be more careful next time”, because humans forget sometimes (just like LLMs). That’s a great way of putting it, I’ll remember that one (except when I forget...)

I am pretty sure you will remember it during your next learning review… as soon as you get in that learning review, it is suddenly very easy to remember all the things you forgot to do.
Post reply on HN