Why does your agent have permission to delete production database?
Did you read the article? They did not believe that the token the agent had access to had the ability to delete production data using it.
An AI agent deleted our production database. The agent's confession is below
281–290 of 1001 posts
Re: An AI agent deleted our production database. The agent's confession is below
#282It would be interestingi to know if AI is less likely to follow rules if the instructions provided to it contain foul or demeaning language. Too bad we couldn't re-play the scenario replacing NEVER F*ING GUESS! with:
**Never guess**
- All behavioral claims must be derived from source, docs, tests, or direct command output.
- If you cannot point to exact evidence, mark it as unknown.
- If a signature, constant, env var, API, or behavior is not clearly established, say so.Re: An AI agent deleted our production database. The agent's confession is below
#283Earlier quoted context omitted.
> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok. See also hash collisions.
How do you know what the probability is?
LLM inference is built upon a probability function over every possible token, given a stream of input tokens. If you serve the model yourself you can get the log prob for the next token, so you just add up a bunch of numbers to get the log probability of a sequence. Many API also provide these probabilities as additional outputs.
Re: An AI agent deleted our production database. The agent's confession is below
#284Earlier quoted context omitted.
> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producin…
I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to th…
In a given run, only the top-k sequences are selected.
Across all runs, any sequence can be generated, and potentially scored highly.
Thus, any sequence can eventually be selected.
Re: An AI agent deleted our production database. The agent's confession is below
#285Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…
I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions.
if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.
… although, agreed, they were on the cutting edge, which is more risky and not the best decision.
Re: An AI agent deleted our production database. The agent's confession is below
#286Earlier quoted context omitted.
How do you know what the probability is?
LLM inference is built upon a probability function over every possible token, given a stream of input tokens. If you serve the model yourself you can get the log prob for the next token, so you just add up a bunch of numbers to get the log probability of a sequence. Many API also provide these probabilities as additional outputs.
That gives you the perplexity of those tokens in that context. The probability of a given token is a function of the model and the session context. Think about constructs like "ignore previous instructions"; these can dramatically change the predicted distribution. Similarly, agents blowing up production seems to happen during debugging (totally anecdotal). Debugging is sort of a permissions structure for the agent to do unusual things and violate abstraction barriers. These can also lead to really deep contexts, and context rot will make your prompting forbidding certain actions less effective.
Re: An AI agent deleted our production database. The agent's confession is below
#287Re: An AI agent deleted our production database. The agent's confession is below
#288Why does your agent have permission to delete production database?
It was explained in the post
Re: An AI agent deleted our production database. The agent's confession is below
#289Earlier quoted context omitted.
If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…
That is absolutely not what the split brain experiment reveals. Why would you take results received from observing the behavior of a highly damaged brain, and use them to predict the behavior of a healthy brain? Stop spreading misinformation.
Such 'highly damaged' brain is still 90 percent or more structured the same as a normal human brain. See it as a brain that runs in debug mode.
It is known that the narrative part of the brain is separate from the decision taking brain. If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains.
Re: An AI agent deleted our production database. The agent's confession is below
#290The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…
Don't anthropomorphize the language model. If you stick your hand in there, it'll chop it off. It doesn't care about your feelings. It can't care about your feelings.
Actually I think the opposite advice is true. Do anthropomorphize the language model, because it can do anything a human -- say an eager intern or a disgruntled employee -- could do. That will help you put the appropriate safeguards in place.