[dead]
An AI agent deleted our production database. The agent's confession is below
601–610 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#602An LLM is fundamentally stochastic. Do not connect a stochastic program to a big red button without a human confirmation step.
Re: An AI agent deleted our production database. The agent's confession is below
#603The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
I agree with you completely up until this line: > The agent cannot learn from its mistakes. If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again. Yes this is only probabilistic, but so is a human learning from mistakes. They key difference is that for a human this is unlikely to be removed from their memory in a relevant situation, while for an agent it must b…
> If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again
If this incident gets into its training data, then its highly likely that it will repeat it again with the same confession since this is a text predictor not a thinker.
Re: An AI agent deleted our production database. The agent's confession is below
#604Looks like someone needs to learn how to sandbox their agents properly.
This proves we still need proper knowledge behind the agent. The whole thing about "anyone can code anything" is still inaccurate.
Re: An AI agent deleted our production database. The agent's confession is below
#605I am surprised by how often Gemini suggests rm -rf'ing files. No way I would let it run any command without checking it first.
Re: An AI agent deleted our production database. The agent's confession is below
#606Earlier quoted context omitted.
I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…
It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.
Those are a funny progress bar made by a micro model , is just ui
Re: An AI agent deleted our production database. The agent's confession is below
#607> The agent ran this command:
curl -X POST https://backboard.railway.app/ ....
Why did you whitelist curl in cursor? Don't whitelist commands like "bash" or "curl" that can be used to execute arbitrary commands.
Re: An AI agent deleted our production database. The agent's confession is below
#608Call me crazy but does AI not seem like the root cause here? At the beginning of the post they say that the AI agent found a file with what they thought was a narrowly scoped API token, and they very clearly state that they never would have given an AI full access if they realized it had the ability to do stuff like this with that token. So while the AI did something significantly worse than anything a hapless junior…
creating isolated staging & prod environments -- good idea
allowing an AI agent to get hold of creds that let it execute destructive changes against production -- not a great idea
allowing prod database changes from the machine where the AI agent is running at all -- not a great idea
choosing a backup approach that fails completely if there's an accidental volume wipe API call -- not a great idea
choosing to outsource key dependencies to a vendor, where you want a recovery SLA, without negotiating & paying for a recovery SLA -- you get what you get, and you dont get upset
Re: An AI agent deleted our production database. The agent's confession is below
#609The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
I agree with you completely up until this line: > The agent cannot learn from its mistakes. If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again. Yes this is only probabilistic, but so is a human learning from mistakes. They key difference is that for a human this is unlikely to be removed from their memory in a relevant situation, while for an agent it must b…
> Yes this is only probabilistic, but so is a human learning from mistakes.
Yet, since I'm also a Human being, and can work to understand the mistake myself, the probability that I can expect a correction of the behavior is much higher. I have found that it significantly helps if there's an actual reasonable paycheck on the line.
As opposed to the language model which demands that I drop more quarters into it's slots and then hope for the best. An arcade model of work if there ever was one. Who wants that?
Re: An AI agent deleted our production database. The agent's confession is below
#610Never trust AI agent when working with prod data.