Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

921–930 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#921

Post by zimpenfish »

ErroneousBosh wrote: post #805
falcor84 wrote: post #802

Earlier quoted context omitted.

It's not common, but I've personally built APIs where requests for dangerous modifications like this perform a dry run, giving in the response the resources that would be deleted/changed and a random token, which then needs to be provide to actually make the change. The idea was that this would be presented in the UI for the user to confirm, but it should be as useful or more by AI agents. Also, you get the benefit t…

Measure twice cut once seems to be forgotten these days.

"Measure twice, THINK ONCE, cut once" is even better[0].

[0] Why yes, I have measured twice, cut once, and made a right old balls up.

Re: An AI agent deleted our production database. The agent's confession is below

#922

Post by pier25 »

juliansark wrote: post #903

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

> Guy gives non-deterministic software root access, desaster happens.

I agree the guy is an idiot for trusting these AI models.

OTOH AI companies keep running and marketing their services with zero accountability for mistakes.

Re: An AI agent deleted our production database. The agent's confession is below

#923

Post by dpark »

laszlojamf wrote: post #833
postexitus wrote: post #812

Earlier quoted context omitted.

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

> At some point you have to trust vendors.

You absolutely do not. When someone makes an unbelievable claim, such as having magic guardrails for LLMs that prevent dangerous actions (what would that even mean?!), you don’t have to trust that claim.

If you trust someone’s claim without justification, that’s on you.

Re: An AI agent deleted our production database. The agent's confession is below

#925

Post by stanfordkid »

I don't think you can really blame AI agents for this. While I agree the user was using AI irresponsibly, some of the blame does go to Railway for making an API key that allows for all operations to happen from a single key without giving clear warnings on privileges. Clearly this user was shooting from the hip and quickly pasted whatever key they got from Railway into a file somewhere so there is some blame there, but any service that handles hosting infrastructure should provide clear UX warning to users regarding the scoping of it's credentials.

Re: An AI agent deleted our production database. The agent's confession is below

#927

Post by boc »

not_kurt_godel wrote: post #758
boc wrote: post #710

Earlier quoted context omitted.

LLMs can research what a tool does before calling it though - they'll sniff that one out pretty quick. I think the better route is to be honest and say that database integrity is a primary foundation of the company, there's no task worth pursuing that would require touching the database, specifically ask it to think hard before doing anything that gets close to the production data, etc. I run a much lower-stakes vers…

> specifically ask it to think hard before doing anything that gets close to the production data This is recklessly negligent and I would personally not tolerate a coworker or report doing it. What's next, sending long-lived access tokens out over email and asking pretty please for nobody to cc/forward?

As described, there are other failsafes as well. The ultimate being that I keep all code version-controlled, and all databases snapshotted offsite daily/hourly and can rebuild them from a complete delete in fewer than X min.

My broader point is that LLMs are going to need access to these keys whether we like it or not, and until we get extremely scoped API permissions (which would make a ton of sense, but most services aren't there), you have to live a bit on the edge to move quickly.

Re: An AI agent deleted our production database. The agent's confession is below

#928

Post by boc »

Jean-Papoulos wrote: post #760
boc wrote: post #710

Earlier quoted context omitted.

LLMs can research what a tool does before calling it though - they'll sniff that one out pretty quick. I think the better route is to be honest and say that database integrity is a primary foundation of the company, there's no task worth pursuing that would require touching the database, specifically ask it to think hard before doing anything that gets close to the production data, etc. I run a much lower-stakes vers…

"I've put enough safety around the bomb that the bomb is worth using. The other people that exploded just didn't have enough safety but I do !"

More like, I expect this bomb can explode, so I've built contingency plans around it because the cost of not using the tooling is much higher than having downtime for my specific use-case.

Re: An AI agent deleted our production database. The agent's confession is below

#929

Post by JohnMakin »

pier25 wrote: post #922
juliansark wrote: post #903

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

> Guy gives non-deterministic software root access, desaster happens. I agree the guy is an idiot for trusting these AI models. OTOH AI companies keep running and marketing their services with zero accountability for mistakes.

I guess people are finding out the hard way you do sorta need technical people to say, "hey, maybe this isn't a great idea" rather than trusting marketing hype that says technical skills are dead.

Re: An AI agent deleted our production database. The agent's confession is below

#930

Post by jryb »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

Just wanted to say that I share any frustration you may feel at every reply to your comment completely missing the point
Post reply on HN