Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

581–590 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#581

Post by gizmo686 »

zthrowaway wrote: post #525
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

True but there’s nothing stopping a webdev dropping an API key in some wiki somewhere in the corporate intranet and the agent quickly picking that up. Can you scan for that? Sure. But it’s a race to see who wins, the scanner or agent.

Maybe I just haven't worked in enough start ups. But where I have worked, there are a lot of things stopping that. Most people don't have access to any production keys. For those that do, we have policies about how to manage them. Those policies go through audits. Our intranet goes through audits.

A production API key appearing on the wiki would be the second biggest security incident I have seen in almost a decade.

------

On the AI note, despite a massive investment in AI (including on-premesise models), we don't give the AI anything close to full access to the intranet because it is almost unimaginable how to square that with our data protection requirements. If the AI has access to something, you need to assume that all users of that AI have access to it. Even if the user themselves is allowed access with it, they will not be aware that the output is potentially tainted, and may share it with someone or thing that should not have access to it.

Re: An AI agent deleted our production database. The agent's confession is below

#582

Post by d3rockk »

tripleee wrote: post #344
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

"An AI agent deleted our production database" should be "I deleted our production database using AI". You can't blame AI any more than you can blame SSH.

Bingo

Re: An AI agent deleted our production database. The agent's confession is below

#585

Post by cnst »

pdantix wrote: post #556
9dev wrote: post #432

Earlier quoted context omitted.

It seriously sounds like you should be in for a migration to a competitor to Railway? Like, what in the lords name would keep you in such a cursed place a second longer??

i migrated to railway earlier in the year after being on vercel for 3 years. in those 3 years, i don't think i was affected by a single incident. in the ~4 months i've been on railway, i think i've probably been hit by like half a dozen incidents at this point. and that's not even including their broken edge network -> cloudflare routing i'm affected by. was told by staff to just move the deployment closer to me, whi…

I think the problem here is that all of these services are optimising for the biggest "change-at-all-cost" that there could be.

If you have a service that does one thing, and does it good, and provides backwards compatibility, it cannot change every day. But if it doesn't change every day, then it's labelled as "obsolete" by those who go after the latest and greatest. If it just works and doesn't require adapting on every level, then those that are after the resume-driven-development, aren't "learning", and thus, again, those services are "old and obsolete".

But you can't have both the "change" and the "stability", something has got to give.

Re: An AI agent deleted our production database. The agent's confession is below

#587

Post by DiogenesKynikos »

getpokedagain wrote: post #341
gigatree wrote: post #333

Earlier quoted context omitted.

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

We are anthropomorphizing whenever we refer to prompts as instructions to models. They predict text not obey our orders.

> They predict text not obey our orders.

Those are the same thing in this case. The latter is just an extremely reductionist description of the mechanics behind the former.

Re: An AI agent deleted our production database. The agent's confession is below

#588

Post by empath75 »

dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

This is 100% the fault of the people misusing the AI.

Re: An AI agent deleted our production database. The agent's confession is below

#589

Post by gwerbin »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

The author definitely deserves a lot of blame here and clearly doesn't understand AI well enough to have a coherent opinion on AI safety.

But Railway bears some responsibility too because, at least of the author is to be believed, it looks like they provide no safety tools for users, regardless of whether they use AI or not. You should be able to generate scoped API tokens. That's just good practice. A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.

Re: An AI agent deleted our production database. The agent's confession is below

#590

Post by jcgrillo »

rahoulb wrote: post #12
heliumtera wrote: post #7

Someone trusted prod database to an llm and db got deleted. This person should never be trusted with computers ever again for being illiterate

If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too. The LLM broke the safety rules it had been given (…

There is no version of this that is the LLM's "fault" for any definition. This was 100% pilot error. When you fly the plane into the side of a mountain on autopilot, it's pilot error every single time.
Post reply on HN