Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

321–330 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#321

Post by maxbond »

xmodem wrote: post #267
stratos123 wrote: post #236

Earlier quoted context omitted.

I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to th…

So it's only wrong in a technical and pedantic sense. A better phrasing might have been along the lines of "There are many sequences of tokens that will destroy your production database that are within the set of possible outputs"

"Everything that can go wrong, will go wrong" isn't literally true either, some failure modes are mutually exclusive so at most one of them will go wrong. I think that the punchy phrasing and the mental model are both more useful from the standpoint of someone creating/managing agents and that it is true in the sense that any other mental model or rule of thumb is true. It's literally true among spherical cows in a frictionless vacuum and directionally correct in the real world with it's nuances. And most importantly adopting the mental model leads to better outcomes.

But it may be a bad mental model in other contexts, like debugging models. As an extreme example models is that collapse during training become strictly deterministic, eg a language model that always predicts the most common token and never takes into account it's context.

Re: An AI agent deleted our production database. The agent's confession is below

#322

Post by bombcar »

baal80spam wrote: post #57
prewett wrote: post #53

My dad always said "pedestrians have the right of way" every time one crossed the street, but wouldn't let us cross the street when the pedestrian light came on until the cars stopped. When I repeated his rule back to him, he said "you may have the right of way, but you'll still be dead if one hits you". My adult synthesis of this is "it's fine to do something risky, as long as you are willing to take the consequence…

Your dad was a wise man. In my country there is a saying: "Graveyards are full of pedestrians that had the right of way".

“You have the right of way but you can be dead right.”

Re: An AI agent deleted our production database. The agent's confession is below

#323

Post by nkrisc »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics.

Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material.

Barring that, it is entirely possible and even quite likely that your ceiling will collapse on you or someone else some time in the future.

It boggles the mind to let an LLM have access to a production database without having explicit preventative measures and contingency plans for it deleting it.

Re: An AI agent deleted our production database. The agent's confession is below

#324

Post by LPisGood »

antonvs wrote: post #271
amluto wrote: post #214

Earlier quoted context omitted.

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

That’s a feature that other humans impose on whoever’s being held accountable. There’s no reason in principle we couldn’t do the same with agents.

How would you fire an agent? This impacts the company that makes the LLM, but not the agent itself.

Re: An AI agent deleted our production database. The agent's confession is below

#326

Post by jbxntuehineoh »

8ytecoder wrote: post #61
Syntaf wrote: post #40

Earlier quoted context omitted.

"Make no mistakes"

> "NEVER FUCKING GUESS!"

hmmmm ok, what if we add a bit more profanity to that? perhaps some extra exclamation marks? maybe that'll make the agents actually follow the rules?

Re: An AI agent deleted our production database. The agent's confession is below

#327

Post by jumpconc »

prewett wrote: post #53

My dad always said "pedestrians have the right of way" every time one crossed the street, but wouldn't let us cross the street when the pedestrian light came on until the cars stopped. When I repeated his rule back to him, he said "you may have the right of way, but you'll still be dead if one hits you". My adult synthesis of this is "it's fine to do something risky, as long as you are willing to take the consequence…

How about the sorcerer's apprentice?

Re: An AI agent deleted our production database. The agent's confession is below

#328

Post by M_bara »

That is why i insist on 1. Streaming replication whether from RDS or my own DB 2. Db dumps shipped to s3 using write only creds or something like rsync.

Streaming gets you PIT recovery while DB dumps give me daily snapshots stored daily for 14 days.

An aside: 15 or so years ago, a work colleague made a mistake and dropped the entire business critical DB - at a critical internet related company - think of continent wide ip issues. I had just joined as a dba and the first thing I’d done was MySQL bin logging. That thing saved our bacon - the drop db statement had been replicated to slaves so we ended up restoring our nightly backup and replaying the binlogs using sed and awk to extract DML queries. Epic 30 minute save. Moral of the story, have a backup of your backup so you can recover when the recovery fails;)

Re: An AI agent deleted our production database. The agent's confession is below

#329

Post by 4ndrewl »

"This is the agent on the record, in writing."

There's no record for the agent to be on - it's always just a bunch of characters that look plausible because of the immense amount of compute we've put behind these, and you were unlucky.

LLMs get things wrong is what we're forever being told.

And the explanation/confession - that's just more 'bunch of characters' providing rationalisation, not confession.

Post reply on HN