Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

761–770 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#761

Post by vidarh »

tempaccount5050 wrote: post #513
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

I think you might be misinterpreting that. I always understood it to mean that when the two hemispheres can't communicate, they'll make things up about their unknowable motivations to basically keep consciousness in a sane state (avoiding a kernel panic?). I don't think it's clear that this happens when both hemispheres are able to communicate properly. At least, I don't think you can imply that this special case is…

We have no reason to believe it is a special case. The fact that these patients largely functioned normally when you did not create a situation preventing the hemispheres from synchronising suggests otherwise to me. There's no reason to think the ability to just make things up and treat it as if it is truthful recollection would just disappear because there are two halves that can lie instead of just one.

Re: An AI agent deleted our production database. The agent's confession is below

#762

Post by andybak »

qsera wrote: post #732
DiogenesKynikos wrote: post #560

Earlier quoted context omitted.

The same could be said for your brain. LLMs are highly intelligent. Comparing them to spreadsheets is reductionist and highly misleading.

>LLMs are highly intelligen I will tell you why it is not. Intelligence is understanding low level stuff and using it to reason about and understand high level stuff. When LLMs demonstrate "highly intelligent" behavior, like solving a complex math problem (high level stuff), but also simultaneously demonstrate that it does not know how to count (low level stuff that the high level stuff depends on), it proves that it…

You just invented you own definition of intelligence. I'm pretty sure that strategy could also support the opposite conclusion.

Re: An AI agent deleted our production database. The agent's confession is below

#763

Post by port11 »

anon84873628 wrote: post #487
sobellian wrote: post #441

Earlier quoted context omitted.

The 'confession' is a CYA. Honestly the whole story doesn't really make sense - what's a "routine task in our staging environment" that needs a full-blown LLM? That sounds ridiculous to me. The takeaway is we commingled creds to our different environments, we gave an LLM access, and we had faulty backups. But it's totally not our fault.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Railway’s “Ship software peacefully” is a good mantra, and they might want to add more protections around very destructive operations.

There’s a lot of blame to be passed around in this story, including OP’s own ways of working. But I agree with them that such destructive operations shouldn’t be in an MCP, or at least be disabled by default.

Re: An AI agent deleted our production database. The agent's confession is below

#764

Post by delegate »

Claude code deleted the database once for me. It wasn't production, but it did contain data I needed. The good thing was that I made a backup of the database right before running claude. I told it that I made a backup, so it decided to delete the db rather than drop the table.

Why did you delete the database? you were supposed to drop the table !

• You're right, I apologize for that mistake. You said to drop the table, not the entire database. I should have run: DROP TABLE IF EXISTS model_requests; Since you mentioned you backed up the database, you can restore it and I'll run the correct SQL command to drop just the model_requests table.

Re: An AI agent deleted our production database. The agent's confession is below

#765

Post by port11 »

coldtea wrote: post #449
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

Using files called SOUL, CONSTITUTION, and so on seems like it would make it more likely we see LLMs as pseudo-alive. It’s both a diminishing of what makes us human and a betrayal of what LLMs truly are (and should be respected as such).

Re: An AI agent deleted our production database. The agent's confession is below

#766

Post by theologic »

not_kurt_godel wrote: post #348
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

For those who might not know the reference: https://simonwillison.net/2024/Sep/17/bryan-cantrill/ : > Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" –…

You have no idea how thankful that you explained that. I watched the Cantrill video. As somebody that dealt this Oracle, it struck home.

Re: An AI agent deleted our production database. The agent's confession is below

#767

Post by port11 »

gigatree wrote: post #333
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

LLMs are probabilistic. The instructions increase the likelihood of a desired outcome, but not deterministically so.

I don’t understand how you can deploy such a powerful tool alongside your most important code and assets while failing to understand how powerful and destructive an LLM can be…

Re: An AI agent deleted our production database. The agent's confession is below

#768

Post by PeterStuer »

I learned not to trust any vendor's backup and recovery promisess when my partner's hosted website, with a monthly paid backup service, had a storage crash and the backup (that had been billed every month for years) turned out not to exist.

Re: An AI agent deleted our production database. The agent's confession is below

#769

Post by AtNightWeCode »

jiggawatts wrote: post #674
crazygringo wrote: post #313

Earlier quoted context omitted.

Yup, this is bizarre. A top use case for needing a backup is when you accidentally delete the original. You need to be able to delete backups too, of course, but that absolutely needs to be a separate API call. There should never be any single API call that deletes both a volume and its backups simultaneously. Backups should be a first line of defense against user error as well. And I checked the docs -- they're call…

Azure SQL Database did this too for a while until enough companies complained about losing their data and their backups with a single action.

With the difference that best practices in Azure SQL have always been to store your own copies of backups and run the database in some HA/GEO-redundancy mode that blocks deletion.

Re: An AI agent deleted our production database. The agent's confession is below

#770

Post by gloosx »

API token with permissions to delete an entire production database in a file? Cool story, this database was destined to vanish. The system rules never mentioned that it shouldn't run destructive POST requests anyway.

I like how they are trying to find a scapegoat – Cursor failure, Railway's failures etc. Guys, it's YOUR failure, is it so hard to admit?

Post reply on HN