Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

401–410 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#401

Post by crazygringo »

lelanthran wrote: post #340
crazygringo wrote: post #319

As unfortunate as this outcome was, the docs clearly state that you should have a recovery period of 48 hours (strange the post doesn't mention it): > Deletion and Restoration > When a volume is deleted, it is queued for deletion and will be permanently deleted within 48 hours. You can restore the volume during this period using the restoration link sent via email. > After 48 hours, deletion becomes permanent and the…

The question here then, is "is that document correct?" If it is then I don't see how the volume got deleted - the mail was not sent? The company was not reading its mails?

I mean, if the document isn't correct it seems like the post should be explicitly mentioning that.

Because without acknowledging it, it comes across as someone writing a dramatic post who doesn't want to let the details get in the way of a good story.

Re: An AI agent deleted our production database. The agent's confession is below

#403

Post by __d »

I’m sorry to be harsh but this is 100% your fault, and attempting to shift the blame onto Cursor and Railway just doesn’t fly.

The onus is on you to make sure your system uses the APIs in a way that’s right for your business. You didn’t. You used a non-deterministic system to drive an API that has destructive potential. I appreciate that you didn’t expect it to do what it did but that’s just naivety.

You’re reaping what you sowed.

Best of luck with the recovery. I hope your business survives to learn this lesson.

Re: An AI agent deleted our production database. The agent's confession is below

#404

Post by [deleted] »

keeda wrote: post #364
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

Agreed, but the point is, if your system is resilient against an eager intern who has not had the necessary guidance, or an actively hostile disgruntled employee, that inherently restricts the harm an LLM can do. I'm not making the case that LLMs learn like people. I'm making the case that if your system is hardened against things people can do (which it should be, beyond a certain scale) it is also similarly hardene…

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#405

Post by MagicMoonlight »

nh2 wrote: post #367
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> The agent cannot learn from its mistakes. The agent will never produce any output which will help you invoke future agents more safely That is not entirely true: Given that more and more LLM providers are sneaking in "we'll train on your prompts now" opt-outs, you deleting your database (and the agent producing repenting output) can reduce the chance that it'll delete my database in the future.

Actually no, it will increase it. Because it’ll be trained with the deletion command as a valid output.

Re: An AI agent deleted our production database. The agent's confession is below

#408

Post by jmalicki »

mmoll wrote: post #175
jmalicki wrote: post #122

Earlier quoted context omitted.

It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.

It doesn’t mean that these “thoughts” influenced their final decision the way they would in humans. An LLM will tell you a lot of things it “considered” and its final output might still be completely independent of that.

Its output quite literally is not independent, as the "thinking tokens" are attended to by the attention mechanism.

Re: An AI agent deleted our production database. The agent's confession is below

#409

Post by gpm »

keeda wrote: post #364
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

Agreed, but the point is, if your system is resilient against an eager intern who has not had the necessary guidance, or an actively hostile disgruntled employee, that inherently restricts the harm an LLM can do. I'm not making the case that LLMs learn like people. I'm making the case that if your system is hardened against things people can do (which it should be, beyond a certain scale) it is also similarly hardene…

The difference makes the necessary barriers different.

There's benefit to letting a human make and learn from (minor) mistakes. There is no such benefit accrued from the LLM because it is structurally unable to.

There's the potential of malice, not just mistakes, from the human. If you carefully control the LLMs context there is no such potential for the LLM because it restarts from the same non-malicious state every context window.

There's the potential of information leakage through the human, because they retain their memories when they go home at night, and when they quit and go to another job. You can carefully control the outputs of the LLM so there is simply no mechanism for information to leak.

If a human is convinced to betray the company, you can punish the human, for whatever that's worth (I think quite a lot in some peoples opinion, not sure I agree). There is simply no way to punish an LLM - it isn't even clear what that would mean punishing. The weights file? The GPU that ran the weights file?

And on the "controls" front (but unrelated to the above note about memory) LLMs are fundamentally only able to manipulate whatever computers you hook them up to, while people are agents in a physical world and able to go physically do all sorts of things without your assistance. The nature of the necessary controls end up being fundamentally different.

Re: An AI agent deleted our production database. The agent's confession is below

#410

Post by root_axis »

Ultimately, storing secrets on disk was the problem here. Never store secrets on disk. This is software engineering 101. The excuse that "we didn't know the scope of the token's access" is absurd. You knew it was a secret with access to production infrastructure, that's all you need to know.

Their provider only having backups on the same volume as the data is also egregious, but definitely downstream of leaking secrets to an adversary. The poorly scoped secrets are also bad, but not uncommon.

With all that stated... this kind of stuff is inevitable if you have an autonomous LLM statistically spamming commands into the CLI. Over a long enough period of time the worst case scenario is inevitable. I wonder how long it will be before people stop believing that adding a prompt which says "don't do the bad thing" doesn't work?

Post reply on HN