Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

301–310 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#301

Post by xnx »

pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

An LLM will reply with a plausible explanation of why someone would have written the code that it just wrote. Seems about the same.

Re: An AI agent deleted our production database. The agent's confession is below

#303

Post by jumpconc »

amluto wrote: post #214
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

You haven't met certain humans. Not all humans have internal capacity for accountability.

The real meaning of accountability is that you can fire one if you don't like how they work. Good news! You can fire an AI too.

Re: An AI agent deleted our production database. The agent's confession is below

#304

Post by BarryMilo »

lelanthran wrote: post #256
mdavid626 wrote: post #70

Earlier quoted context omitted.

In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.

> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation. That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing. The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"

How are people still deluded enough about this economic system to believe rank implies competence?

Re: An AI agent deleted our production database. The agent's confession is below

#305

Post by nkrisc »

keeda wrote: post #290
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

It is merely a simulacrum of an intern or disgruntled employee or human. It might say things those people would say, and even do things they might do, but it has none of the same motivations. In fact, it does not have any motivation to call its own.

Re: An AI agent deleted our production database. The agent's confession is below

#306

Post by unyttigfjelltol »

amluto wrote: post #214
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

I disagree. They could fire Claude and their legal counsel could pursue claims (if there were any, idk)-- the accountability model is similar. Anthropic probably promised no particular outcome, but then what employee does?

And in the reverse, if a person makes a series of impulsive, damaging decisions, they probably will not be able to accurately explain why they did it, because neither the brain nor physiology are tuned to permit it.

Seems pretty much the same to me.

Re: An AI agent deleted our production database. The agent's confession is below

#307

Post by jbxntuehineoh »

jeremyccrane wrote: post #235
powera wrote: post #35

Earlier quoted context omitted.

He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.

In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?

... because that's how every other cloud provider API works? the AWS console makes you confirm before deleting a bucket; DeleteBucket does not

Re: An AI agent deleted our production database. The agent's confession is below

#309

Post by braebo »

gpm wrote: post #296
keeda wrote: post #290

Earlier quoted context omitted.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

Re: An AI agent deleted our production database. The agent's confession is below

#310

Post by lelanthran »

tee-es-gee wrote: post #199
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

I do think that as service providers we now have a new "attack vector" to be worried about. Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, because generally users won't do such a destructive action via the API or if they do, they likely understand the consequences. Or at the very least don't complain if they do it without reading the docs carefully enough. But n…

> Up to now, having an API that deletes the whole volume, including backups, might have been acceptable,

So? I have those too; the difference is that:

1. The API is ACL'ed up the wazoo to ensure only a superuser can do it.

2. The purging of data is scheduled for 24h into the future while the unlinking is done immediately.

3. I don't advertise the API as suitable for agent interaction.

Post reply on HN