Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

341–350 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#341

Post by getpokedagain »

gigatree wrote: post #333
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

We are anthropomorphizing whenever we refer to prompts as instructions to models. They predict text not obey our orders.

Re: An AI agent deleted our production database. The agent's confession is below

#342

Post by hibouaile »

This is a classic anchoring failure. The LLM read the request, framed the risk space ("looks like cleanup is needed"), and the human didn't challenge that framing before it acted.

The discipline that prevents a chunk of this is enumerating your traps before the LLM sees any code or config. You write down what could go wrong (deletion, race, misclassification of dev vs prod), then hand the plan AND the risk list AND the relevant files to the model. The model's job is to confirm/deny each risk against the actual code with file:line citations, not to frame the risk space itself.

Pre-implementation. Anchoring defense. The opposite of "vibe coding."

Re: An AI agent deleted our production database. The agent's confession is below

#343

Post by whstl »

braebo wrote: post #309
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

Which it will start ignoring after two or three messages in the session.

Re: An AI agent deleted our production database. The agent's confession is below

#344

Post by tripleee »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

"An AI agent deleted our production database" should be "I deleted our production database using AI".

You can't blame AI any more than you can blame SSH.

Re: An AI agent deleted our production database. The agent's confession is below

#345

Post by dpark »

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.”

You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops, I ran DROP TABLE on the production database” story. There’s no excuse for enabling a system where this can happen and it’s unacceptable to shift blame when faced with the reality that this is exactly what you did.

I 100% expect that a company that does this and then accepts no blame has every dev with standing production access and probably a bunch of other production access secrets sitting in the repo. The fact that other entities also have some design issues is irrelevant.

Re: An AI agent deleted our production database. The agent's confession is below

#347

Post by Quarrelsome »

braebo wrote: post #309
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

and you'll blow the context over time and send to the LLM sanitorium. It doesn't fit like the human brain can.

If a junior fucks production that will have extroadinary weight because it appreciates the severity, the social shame and they will have nightmares about it. If you write some negative prompt to "not destroy production" then you also need to define some sort of non-existing watertight memory weighting system and specify it in great detail. Otherwise the LLM will treat that command only as important as the last negative prompt you typed in or ignore it when it conflicts with a more recent command.

Re: An AI agent deleted our production database. The agent's confession is below

#348

Post by not_kurt_godel »

xmodem wrote: post #273
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

For those who might not know the reference: https://simonwillison.net/2024/Sep/17/bryan-cantrill/:

> Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" – lawnmower doesn’t give a shit about you, lawnmower can’t hate you. Don’t anthropomorphize the lawnmower. Don’t fall into that trap about Oracle.

> — Bryan Cantrill

Re: An AI agent deleted our production database. The agent's confession is below

#349

Post by atoav »

Ah? Running random code on a machine that can potentially delete production data is a fucking stupid idea.

Sorry to be that guy, but: LLMs agents are experimental by this point. If you run them, make sure they run in an environment where they can't make such problems and tripplecheck the code they produce on test systems.

That is due diligence. Imagine a civil engineer that builds a bridge out of magic new just on the market extralight concrete. Without tests. And then the bridge collapses. Yeah, don't be that person. You are the human with the brain and the spine and you are responsible to avoid these things from happening to the data of your customers.

Also: just restore the backup? Or do we not have a backup? If so, there is really no mercy. Backups are the bare minimum since decades now.

Re: An AI agent deleted our production database. The agent's confession is below

#350

Post by collinmcnulty »

braebo wrote: post #309
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

And the memento guy had tattoos of key information. That didn’t make it so he didn’t have memory loss.
Post reply on HN