Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

491–500 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#491

Post by hunterpayne »

oskarkk wrote: post #259
gwerbin wrote: post #131

Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…

> what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token It sounds like the token the author created just didn't have any scope, it had full permissions. From the post: > Tokens are not scoped by operation, by environment, or by resource at the permission level. There is…

Then you need to reread the article. The author made a key for the LLM that didn't have permissions to delete a volume. The agent then found ANOTHER key with those permissions and used that instead.

Re: An AI agent deleted our production database. The agent's confession is below

#492

Post by protocolture »

grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

"I literally requested no screw ups, and this is a screw up"

I bet these people are bad at managing humans too.

Re: An AI agent deleted our production database. The agent's confession is below

#493

Post by naikrovek »

coldtea wrote: post #450
hdndjsbbs wrote: post #448

Earlier quoted context omitted.

taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.

That is not that strong an argument as it seems, because we too might very well be "a series of weights for probable next tokens". The main difference is the training part and that it's always-on.

We are much more than weights which output probable next tokens.

You are a fool if you think otherwise. Are we conscious beings? Who knows, but we’re more than a neural network outputting tokens.

Firstly, and most obviously, we aren’t LLMs, for Pete’s sake.

There are parts of our brains which are understood (kinda) and there are parts which aren’t. Some parts are neural networks, yes. Are all? I don’t know, but the training humans get is coupled with the pain and embarrassment of mistakes, the ability to learn while training (since we never stop training, really), and our own desires to reach our own goals for our own reasons.

I’m not spiritual in any way, and I view all living beings as biological machines, so don’t assume that I am coming from some “higher purpose” point of view.

Re: An AI agent deleted our production database. The agent's confession is below

#495

Post by hunterpayne »

filoleg wrote: post #209
alastairr wrote: post #25

If it's real this is a terrible thing to have happen. However the moral of this story is nothing to do with AI and everything to do with boring stuff like access management.

^This. One of the top replies on twitter to the OP can be boiled down to "you treat AI as a junior dev. Why would you give anyone, let alone a junior dev, direct access to your prod db?" And yeah, I fully agree with this. It has been pretty much the general consensus at any company I worked at, that no person should have individual access to mess with prod directly (outside of emergency types of situations, which hav…

The LLM didn't have a prod key. It found a prod key in the source base and used that instead of the key it was given.

Re: An AI agent deleted our production database. The agent's confession is below

#497

Post by rcxdude »

theshrike79 wrote: post #383
jumpconc wrote: post #311

Earlier quoted context omitted.

All my backups are inside the same universe as what is being backed up. A boundary must be drawn somewhere and this is one of many reasonable boundaries. As I understand it, the backup isn't "inside" the volume but is attached to it so that deleting the volume deletes the backups.

Can we at least agree to draw the line so that if a single call can delete the live data AND all backups, they shouldn't be called "backups", but rather snapshots?

I would also say that if your backup is controlled by the same third party as the primary, it's not a backup.

Re: An AI agent deleted our production database. The agent's confession is below

#498

Post by linsomniac »

This is your reminder to set up canary tokens: https://canarytokens.org/nest/

I had a token I set up 3 years ago for AWS that I hadn't used. I was recently doing something with Claude and was asking it to interact with our AWS dev environment. I was watching it pretty closely and saw it start to struggle (I forget what exactly was going on), and I was >50% likely it was going to hit my canary token. Sure enough, a few minutes later it did and I got an email. Part of why I let it continue to cook was that I hadn't tested my canary in ~3 years.

Re: An AI agent deleted our production database. The agent's confession is below

#499

Post by hunterpayne »

Rekindle8090 wrote: post #51
comrade1234 wrote: post #49

Some of this stuff is so embarrassing. Why would you even post this online?

Because its fake and its marketing

No, what is fake are all the people defending the LLM. Wait...that means I'm replying to a bot

Re: An AI agent deleted our production database. The agent's confession is below

#500

Post by sombragris »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

The whole use of AI agents in this context reminds me of the movie "War Games"

  > A strange game.
  > The only winning move is
  > not to play.
Post reply on HN