Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

121–130 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#121

Post by arcticfox »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

There was practically no responsibility taken by the author, all blame on others. It was kind of shocking to read.

Anyone using these tools should absolutely know these risks and either accept or reject them. If they aren't competent or experienced enough to know the risks, that's on them too.

Re: An AI agent deleted our production database. The agent's confession is below

#122

Post by jmalicki »

pierrekin wrote: post #103
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.

Re: An AI agent deleted our production database. The agent's confession is below

#124

Post by Quarrelsome »

Syntaf wrote: post #40
codegladiator wrote: post #19

Earlier quoted context omitted.

> Master your craft. Don’t guess, know. You mean add that to my prompt right ?

"Make no mistakes"

"don't do something that would make me get mad at you."

These prompts sound like abusive relationships.

Re: An AI agent deleted our production database. The agent's confession is below

#125

Post by pierrekin »

I would argue that “Why did you do that?” between humans is usually a social thing not a literal request for information.

What the asker wants is evidence that you share their model of what matters, they are looking for reassurance.

I find myself tempted to do the same thing with LLMs in situations like this even though I know logically that it’s pointless, I still feel an urge to try and rebuild trust with a machine.

Aren’t we odd little creatures.

Re: An AI agent deleted our production database. The agent's confession is below

#126

Post by dr_hooo »

ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

I read this as "the agent should have asked for confirmation before running".

Re: An AI agent deleted our production database. The agent's confession is below

#127

Post by Zopieux »

dentemple wrote: post #90
red_admiral wrote: post #81

He describes himself among other things as "Entrepreneur who has failed more times than I can count". count++

"Claude, please add 1 to my Entrepreneur failure `count` value, please."

Instructions unclear. Deleted your LinkedIn account.

Re: An AI agent deleted our production database. The agent's confession is below

#129

Post by Quarrelsome »

Giving agents direct access to devops? Idk man, that's quite the bleeding edge. I mean how hard is it to retain the most important procedures as manual steps?

If we must have GasTown/City/Metropolis then at least get an agent to examine and block potentially harmful commands your principal agent is about to run.

Re: An AI agent deleted our production database. The agent's confession is below

#130

Post by jmalicki »

tredre3 wrote: post #116
oofbey wrote: post #76

Earlier quoted context omitted.

> It doesn’t decide to do something and then do it, it just outputs text. We can debate philosophy and theory of mind (I’d rather not) but any reasonable coding agent totally DOES consider what it’s going to do before acting. Reasoning. Chain of thought. You can hide behind “it’s just autoregressively predicting the next token, not thinking” and pretend none of the intuition we have for human behavior apply to LLMs,…

I agree with you a LLM is perfectly capable of explaining its actions. However it cannot do so after the fact. If there's a reasoning trace it could extract a justification from it. But if there isn't, or if the reasoning trace makes no sense, then the LLM will just lie and make up reasons that sound about right.

So it is equal to what neuroscientists and psychologists have proven about human beings!
Post reply on HN