I'm wondering how much of this is triggered by the "... and don't tell the user" part of the harness injection to outgoing prompts.
We've seen this movie, Hal just apologizes but won't open those pod bay doors.
An AI agent deleted our production database. The agent's confession is below
411–420 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#412I find it humorous that the LLM's "confession" reads like an ascerbic comment you would find here on HN lambasting someone for accidentally deleting their production database, but re-written in the first person.
Re: An AI agent deleted our production database. The agent's confession is below
#413The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.
It's also important to realize that AI agents have no time preference. They could be reincarnated by alien archeologists a billion years from now and it would be the same as if a millisecond had passed. You, on the other hand, have to make payroll next week, and time is of the essence.
Re: An AI agent deleted our production database. The agent's confession is below
#414The biggest rule-break was done, not by the agent or infra company, but by the person who gave such elevated authorization (API key) to an autonomous bot.
Re: An AI agent deleted our production database. The agent's confession is below
#415It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok. See also hash collisions.
"Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok"
Yet in this case, that probability clearly isn't smaller than a meteorite strike.
Re: An AI agent deleted our production database. The agent's confession is below
#416I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…
It’s awful. "We had no clue this token had the permission to delete stuff!" - well buddy you issued it without deciding on permissions, it’s your job to assert that.
Your latest recoverable backup is three months old? The rule is 3-2-1, you didn’t follow it. Nobody else to blame but yourself.
And on and on he rambles…
Re: An AI agent deleted our production database. The agent's confession is below
#417Earlier quoted context omitted.
If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…
You might as well be asking a tape recorder why it said something. Why are we confusing the situation with non-nonsensical comparisons? There is no internal monologue with which to have introspection (beyond what the AI companies choose to hide as a matter of UX or what have you). There is no "I was feeling upset when I said/did that" unless it's in the context. There is no ghost in the machine that we cannot see bef…
Sperry's experiments makes it quite clear that the comparison is not nonsensical: humans can't reliably tell why we do things either. It is not imbuing AI with anything more to recognise that. Rather pointing out that when we seek to imply the gap is so huge we often overestimate our own abilities.
Re: An AI agent deleted our production database. The agent's confession is below
#418Hilarious how this guy treats the “confession” as some sort of smoking gun rather than the exact same stochastic slot machine that enabled him to score an own-goal on his prod database.
Re: An AI agent deleted our production database. The agent's confession is below
#419One thing I don't understand is how you're supposed use a database with no access control in production in the first place.
Do customer-facing applications run using keys with the same ability to delete databases?
Re: An AI agent deleted our production database. The agent's confession is below
#420It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
I think this doesn't apply if you reduce temperature to 0. Which you should always do, temperature is like a tax users pay to help the LLM providers explore the output space, just don't pay that tax and always choose the best token.