Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#541

Post by zaphirplane »

narrator wrote: post #413
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

It's also important to realize that AI agents have no time preference. They could be reincarnated by alien archeologists a billion years from now and it would be the same as if a millisecond had passed. You, on the other hand, have to make payroll next week, and time is of the essence.

Well there were a bunch of articles about resuming a parked session relating to degradation of capabilities and high token usage. Ironic Another example of attempting to treat the LLM as an AI

Re: An AI agent deleted our production database. The agent's confession is below

#542

Post by simonh »

MagicMoonlight wrote: post #405
nh2 wrote: post #367

Earlier quoted context omitted.

> The agent cannot learn from its mistakes. The agent will never produce any output which will help you invoke future agents more safely That is not entirely true: Given that more and more LLM providers are sneaking in "we'll train on your prompts now" opt-outs, you deleting your database (and the agent producing repenting output) can reduce the chance that it'll delete my database in the future.

Actually no, it will increase it. Because it’ll be trained with the deletion command as a valid output.

Exactly. It’s just giving the LLM a token pattern, and it’s designed to reproduce token patterns. That’s all it does. At some point generating a token pattern like that again is literally it’s job.

Re: An AI agent deleted our production database. The agent's confession is below

#543

Post by gessha »

keeda wrote: post #290
xmodem wrote: post #273

Earlier quoted context omitted.

Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.

Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.

You don't anthropomorphize a table saw, you just don't put your hand in there.

Re: An AI agent deleted our production database. The agent's confession is below

#544

Post by 827a »

fathermarz wrote: post #470
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

Completely agree. This is a harness problem, not a model problem. The model is rarely the issue these days

More-so an environment problem. An agent doing staging or development tasks should never be able to get access to prod API credentials, period. Agents which do have access to prod should have their every interaction with the outside world audited by a human.

Re: An AI agent deleted our production database. The agent's confession is below

#545

Post by leptons »

maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

There is no way in hell I would give an LLM direct access to a database to write whatever query it wants. Just no way.

I'll create some safe APIs that I give the LLM access to where it can interact with a limited set of things the database can do, at most.

Re: An AI agent deleted our production database. The agent's confession is below

#547

Post by smj-edison »

crazygringo wrote: post #313
hu3 wrote: post #140

The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.

Yup, this is bizarre. A top use case for needing a backup is when you accidentally delete the original. You need to be able to delete backups too, of course, but that absolutely needs to be a separate API call. There should never be any single API call that deletes both a volume and its backups simultaneously. Backups should be a first line of defense against user error as well. And I checked the docs -- they're call…

Plus backups should be time gated, where the software physically blocks you from removing backups for X days.

Re: An AI agent deleted our production database. The agent's confession is below

#548

Post by kibwen »

anon84873628 wrote: post #482
coldtea wrote: post #449

Earlier quoted context omitted.

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

They should at least stop responding in the first person.

We need to make like Japanese and come up with some neo-first-person-pronouns for bots to use to refer to themselves.

Re: An AI agent deleted our production database. The agent's confession is below

#549

Post by coalstartprob »

shiandow wrote: post #507
lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

For a company that puts DO NOT FUCKING GUESS in their instructions they made a heck of a lot of assumptions - assume tokens are scoped (despite this apparently not even being an existing feature?) - assume an LLM didn't have access - assume an LLM wouldn't do something destructive given the power - assume backups were stored somewhere else (to anyone reading, if you don't know where they are, you're making the same a…

[dead]

Re: An AI agent deleted our production database. The agent's confession is below

#550

Post by coldtea »

smrtinsert wrote: post #524
coldtea wrote: post #449

Earlier quoted context omitted.

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

> The problem is millions of years of evolutionary wiring makes us see it as alive Maybe for laymen, but I would think most technologists should understand that we're working with the output of what is effectively a massive spreadsheet which is creating a prediction.

The thing with evolutionary wiring is that it doesn't matter if you're layman or "technologist". The technologist part is just a small layer on top of very thick caveman/animal insticts and programming.

That's why a technologist can, just as easily as any layman, get addicted to gambling, or do crazy behaviors when attracted by the opposite sex.

Post reply on HN