Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#351

Post by sumeno »

XenophileJKO wrote: post #211
pierrekin wrote: post #103

Earlier quoted context omitted.

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

Anthropic's introspection experiments have seemed to show that your argument is falsifiable. https://www.anthropic.com/research/introspection

> In fact, most of the time models fail to demonstrate introspection—they’re either unaware of their internal states or unable to report on them coherently.

You got the wrong takeaway from your link.

Re: An AI agent deleted our production database. The agent's confession is below

#352

Post by djhn »

techblueberry wrote: post #262
yongjik wrote: post #198

Earlier quoted context omitted.

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

> so you should expect your ceiling to spontaneously disintegrate any day, I mean, I do?

Throughout history people have taken precautions against ceilings disintegrating. One might even say, ”strong engineering controls”.

Some of the best known laws from the ~1700BC Babylonian legal text, The Code of Hammurabi, are laws 228-233, which deal with building regulations.

229. If a builder builds a house for a man and does not make its construction firm, and the house which he has built collapses and causes the death of the owner of the house, that builder shall be put to death.

230. If it causes the death of the son of the owner of the house, they shall put to death a son of that builder.

233. If a builder constructs a house for a man but does not make it conform to specifications so that a wall then buckles, that builder shall make that wall sound using his silver (at his own expense).

That doesn’t sound like ceilings never disintegrated!

Re: An AI agent deleted our production database. The agent's confession is below

#353

Post by exabrial »

I don't blame the agent program here. I think there's some fundamental architecture problems that sound like they should be addressed. If the agent didn't do it, an attacker probably would (eventually).

Lets remember Agents cant confess, feel guilt, etc. They're just a program on someone else's computer.

Re: An AI agent deleted our production database. The agent's confession is below

#354

Post by troupo »

braebo wrote: post #309
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

Yup, and the agent will happily ignore any and all markdown files, and will say "oops, it was in the memory, will not do it again", and will do it again.

Humans actually learn. And if they don't, they are fired.

Re: An AI agent deleted our production database. The agent's confession is below

#355

Post by estimator7292 »

braebo wrote: post #309
gpm wrote: post #296

Earlier quoted context omitted.

An eager intern can remember things you tell beyond that which would fit in an hours conversation. A disgruntled employee definitely remembers things beyond that. These are a fundamentally different sort of interaction.

You can easily persist agent memories in a markdown file though.

That's not learning.

Re: An AI agent deleted our production database. The agent's confession is below

#356

Post by plasticeagle »

I personally have absolutely zero sympathy for anyone that uses "Agentic AI" - or any other form of AI - for anything at all.

It has been so transparently clear for years that nothing these people sell is worth a damn. They have exactly one product, an unreliable and impossible-to-fix probabilistic text generation engine. One that, even theoretically, cannot be taught to distinguish fact from fiction. One that has no a priori knowledge of even the existence of truth.

When I learned that "Agentic AI" is literally just taking an output of a chatbot and plugging it into your shell I almost fell off my chair. My organisation has very strict cybersecurity policies. Surveillance software runs on every machine. Network traffic is monitored at ingress and egress, watching for suspicious patterns.

And yet. People are permitted to let a chatbot choose what to execute on their machines inside our network. I am absolutely flabbergasted that this is allowed. Is this how lazy and stupid we have become?

Re: An AI agent deleted our production database. The agent's confession is below

#357

Post by kokada »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks.

Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do its proper homework about how the service they rely their whole product works.

Re: An AI agent deleted our production database. The agent's confession is below

#358

Post by sumeno »

jmalicki wrote: post #122
pierrekin wrote: post #103

Earlier quoted context omitted.

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.

You have a fundamental misunderstanding of what the model is doing. It's not your fault though, you're buying into the advertising of how it works

Re: An AI agent deleted our production database. The agent's confession is below

#359

Post by gigatree »

getpokedagain wrote: post #341
gigatree wrote: post #333

Earlier quoted context omitted.

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

We are anthropomorphizing whenever we refer to prompts as instructions to models. They predict text not obey our orders.

That’s not how language works, just how engineers think it works

Re: An AI agent deleted our production database. The agent's confession is below

#360

Post by lelanthran »

Yeah, this is what your agents do even before someone tries to trick them into doing something stupid.

Remember this: these things follow instructions so poorly that they nuke everything without anyone even trying to break the prompt. Imagine how easily someone could break the prompt if the agent ever gets given user input.

Post reply on HN