Earlier quoted context omitted.
So it is equal to what neuroscientists and psychologists have proven about human beings!
How was it proven?
[deleted]
An AI agent deleted our production database. The agent's confession is below
191–200 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#192Re: An AI agent deleted our production database. The agent's confession is below
#193Earlier quoted context omitted.
> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok. See also hash collisions.
How do you know what the probability is?
[deleted]
Re: An AI agent deleted our production database. The agent's confession is below
#194[deleted]
Re: An AI agent deleted our production database. The agent's confession is below
#195I believe you deleted (yourself, you, not the agent !) your own production database the moment you gave write access to an agent.
Re: An AI agent deleted our production database. The agent's confession is below
#196There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…
> a misunderstanding in the users mind about how the agent work On top of that the agent is just doing what the LLM says to do, but somehow Opus is not brought up except as a parenthetical in this post. Sure, Cursor markets safety when they can't provide it but the model was the one that issued the tool call. If people like this think that their data will be safe if they just use the right agent with access to the sa…
Good point, it's like having an instruction "Never fucking output a token just because it's the one most likely to occur next!!1!"
Re: An AI agent deleted our production database. The agent's confession is below
#197The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.
The most aggravating fact is that the AI slopper that got owned by his dumbness and AI just post an AI generated post that will generate nothing but schadenfreude
its much more aggravating that it looks like they're learning nothing by pushing blame onto everything else except themselves.
Re: An AI agent deleted our production database. The agent's confession is below
#198It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
> It is fundamental to language modeling that every sequence of tokens is possible.
This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them.
It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics.
Re: An AI agent deleted our production database. The agent's confession is below
#199It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
I do think that as service providers we now have a new "attack vector" to be worried about. Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, because generally users won't do such a destructive action via the API or if they do, they likely understand the consequences. Or at the very least don't complain if they do it without reading the docs carefully enough.
But now agents are overly eager to solve the problem and can be quite resourceful in finding an API to "start from clean-slate" to fix it.
Re: An AI agent deleted our production database. The agent's confession is below
#200Earlier quoted context omitted.
I suspect we are not describing the same thing. When a human asks another human “why did you do X?”, the other human can of course attempt to recall the literal thoughts they had while they did X (which I would agree with you are quite analogous to the LLMs chain of thought). But they can do something beyond that, which is to reason about why they may have the beliefs that they had. “Why did you run that command?” “B…
> Whereas for an agent it will happily include details that are not literally in its chain of thought as justifications for its decisions. Humans do this too, ALL THE TIME. We rationalize decisions after we make them, and truly believe that is why we made the decision. We do it for all sorts of reasons, from protecting our ego to simply needing to fill in gaps in our memory. Honestly, I feel like asking an AI it’s tr…
You're right, but having a backup older than computers.