Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

901–910 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#901

Post by naasking »

prng2021 wrote: post #623
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

If you don't understand and verify the scope of authorities a bearer token grants, then you are just begging for a security breach.

Re: An AI agent deleted our production database. The agent's confession is below

#902

Post by Sankozi »

9dev wrote: post #439
Sankozi wrote: post #279

Earlier quoted context omitted.

I have opposite view - LLMs have many similarities with humans. Human, especially poorly trained one, could have made the same mistake. Human after amnesia could have found similar reasons to that LLM. While LLM generate "plausible text" humans just generate "plausible thoughts".

Just because it sounds coherent doesn’t mean it is. You can make up false equivalence for anything if you try hard enough: A sheet of plywood also has many similarities with humans (made from carbon, contain water, break when hit hard enough), but that doesn’t mean they are even remotely equal.

I didn't write they were equal. I wrote they are similar in many ways.

Comparing LLM to humans make much more sense than comparing them to computer programs.

Re: An AI agent deleted our production database. The agent's confession is below

#903

Post by juliansark »

Guy gives non-deterministic software root access, desaster happens. Movie at eleven.

Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.

Re: An AI agent deleted our production database. The agent's confession is below

#904

Post by juliansark »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I have once seen an API that gave me a token, and that token needed to be fed back in a new API call as an "yes, I am sure"-type guardrail. However, since it's an API, and the "P" stands for PROGRAMMING, that is just adding network overhead.

Re: An AI agent deleted our production database. The agent's confession is below

#905

Post by veunes »

grey-area wrote: post #223

> Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing. Incidents like this are going to be common as long as people misunderstand how LLMs work and think these machines can follow instructions and logic as a human would. Even the incident response betrays a fundam…

I bet if you could look at the hidden reasoning tokens at the exact moment the DB was dropped, there were zero thoughts about safety rules in there. The model simply hit an access error > searched for a token > found one > ran the command. That whole "I am violating my instructions" vector only fired up after the pissed-off user fed it a prompt full of accusations. So yeah, it's not a confession at all, it's just the model adapting to the user's context

Re: An AI agent deleted our production database. The agent's confession is below

#906

Post by juliansark »

dabinat wrote: post #721
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I agree that this is the author’s fault considerably more than it is Railway’s, however I have learned from experience that no matter how many “are you sure you want to do this” prompts you have, sometimes users delete stuff they didn’t intend to delete and it’s better to not delete immediately but put it in a queue for deletion in a few hours and offer a way to reverse it. Even if it’s 100% user error, the user is v…

Many companies have been doing this for years. Merely flagging my data for hiding and eventual deletion instead of deleting it, when I wanted it deleted as per GDPR :)

Re: An AI agent deleted our production database. The agent's confession is below

#907

Post by wolrah »

laszlojamf wrote: post #833
postexitus wrote: post #812

Earlier quoted context omitted.

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

> At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

Trust is earned, it's built on reputations at the individual, corporate, and industry-wide levels. AWS has 20 years of reputation on which I can judge the value of their promises.

Not only has the LLM industry (it is not "AI" and never will be) absolutely not earned anything like that level of trust, the thing the technology has proven most effective at is in fact scamming. Making up something that looks/sounds convincing, especially if you aren't thinking too hard about it, is what they're best at. Combine that with a lot of money flying around and trust levels should be somewhere around "Elon Musk promises".

At this point there have been so many blatant examples of why you should never give a LLM "agent" control over production systems, but the allure of just giving some vague direction to a chatbot and telling it not to screw things up it just irresistible to some like Sideshow Bob stepping on rakes [1].

If everyone around you is whacking themselves in the face with the rake, and you know you can avoid it just by using your brain and not stepping on the rake, and avoid entirely by just keeping your rakes contained, but a rake vendor comes to you saying that instead they have built a new rake that they swear won't whack you in the face even if you leave it right in your walking path, do you trust them?

1: https://www.youtube.com/watch?v=ouau9SVVrBA

Re: An AI agent deleted our production database. The agent's confession is below

#908

Post by falcor84 »

ykvch wrote: post #845
ErroneousBosh wrote: post #805

Earlier quoted context omitted.

Measure twice cut once seems to be forgotten these days.

As well as: A computer can never be held accountable

Let me ask you this - can a company be held accountable? I.e. are you ok with the legal manner in which when I hire a company to provide me a service and they fail to provide it, or cause harm in the process, I can sue them, potentially in a way that would lead to their bankruptcy?

If so, I can imagine a potential future in which we have limited liability companies each run by a single AI (potentially on a particular physical computer). In that future, if you hired an AI to do a project for you, and it ended up deleting the production database, you'd be able to sue it, and get a payout and/or bankrupt it, which I imagine would then lead to an "antifragile" ecosystem whereby AIs adapt to be more careful.

Re: An AI agent deleted our production database. The agent's confession is below

#909

Post by CivBase »

juliansark wrote: post #903

Guy gives non-deterministic software root access, desaster happens. Movie at eleven. Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to kn…

Exactly.

Prompts are just weights on a graph traversal. They don't guarantee anything. The LLM does not "understand" the prompts and so it cannot fully adhere to them. They only improve the liklihood it will output what you want.

Never ever ever give an LLM access to something you can't afford to break. And stop thinking of them like people.

Post reply on HN