Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#751

Post by lmm »

cmiles74 wrote: post #190
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

None of the developers that I’ve worked with have had the hemispheres of their brains severed. I suspect this is pretty rare in the field.

> None of the developers that I’ve worked with have had the hemispheres of their brains severed.

But are their explanations for how they behaved any more compelling than those of people who have? If so, why?

Re: An AI agent deleted our production database. The agent's confession is below

#752

Post by not_kurt_godel »

skeledrew wrote: post #475
not_kurt_godel wrote: post #348

Earlier quoted context omitted.

For those who might not know the reference: https://simonwillison.net/2024/Sep/17/bryan-cantrill/ : > Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" –…

404 on that link.

Works fine for me but maybe try https://web.archive.org/web/20260426213142/https://simonwill...

Re: An AI agent deleted our production database. The agent's confession is below

#753

Post by lmm »

fc417fc802 wrote: post #217
fragmede wrote: post #189

Earlier quoted context omitted.

Claude code and codex both hide the Chain of Thought (CoT) but it's just words inside a set of tags and the agent within the same session has access to that plaintext.

Those are just words inside arbitrary tags, they aren't actually thoughts. Think of it as asking the model to role play a human narrating his internal thought process. The exercise improves performance and can aid in human understanding of the final output but it isn't real.

What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?

Re: An AI agent deleted our production database. The agent's confession is below

#754

Post by kaibee »

Jensson wrote: post #614
lifis wrote: post #595

Earlier quoted context omitted.

You don't seem to realize that humans also work this way. If you ask a human why they did something, the answer is a guess, just like it is for an LLM. That's because obviously there is no relationship between the mechanisms that do something and the ones that produce an explanation (in both humans and LLMs). An example of evidence from Wikipedia, "split brain" article: The same effect occurs for visual pairs and rea…

Most humans don't have split brains, and without split brains you have quite a bit of insight into the thoughts in your brain. Its not perfect but its better than nothing, LLM have nothing since there is no mechanism for them to communicate forward except the text they read.

> Most humans don't have split brains, and without split brains you have quite a bit of insight into the thoughts in your brain. Its not perfect but its better than nothing, LLM have nothing since there is no mechanism for them to communicate forward except the text they read.

I can't prove it but this is almost certainly one of those things that is uh, less than universal in the population.

Re: An AI agent deleted our production database. The agent's confession is below

#755

Post by dcposch »

The AI part of this is a red herring. This is above all a big devops failure.

Three takeaways:

1. TEST YOUR BACKUPS. If you have not confirmed that you can restore, then you don’t have backup. If the backups are in the same place as your prod DB, you also don’t have backup.

2. Don’t use Railway. They are not serious.

3. Don’t rely on this guy. The entire postmortem takes no accountability and instead includes a “confession” from Cursor agent. He is also not serious.

4. See #1.

Running a single bad command will happen sometimes, whether by human or machine. If that’s all it takes to perma delete your service then what you have is a hackathon project, not a business.

Re: An AI agent deleted our production database. The agent's confession is below

#756

Post by vidarh »

jayd16 wrote: post #533
vidarh wrote: post #417

Earlier quoted context omitted.

Sperry's experiments makes it quite clear that the comparison is not nonsensical: humans can't reliably tell why we do things either. It is not imbuing AI with anything more to recognise that. Rather pointing out that when we seek to imply the gap is so huge we often overestimate our own abilities.

It is non-sensical because you're simply bringing in comparisons without anything linking the two. You might as well be talking about how oranges, and bicycles think as well as that is just as relevant as how humans think in this discussion. In fact, talking about "thinking" at all is already the wrong direction to go down when trying to triage an incident like this. "Do not anthropomorphize the lawnmower" applies to…

The thing linking the two is that neither are able to accurately introspect and explain the actual reason why they made a decision.

If thinking is the wrong direction to go down, then it is also the wrong direction to go down when talking about humans.

Re: An AI agent deleted our production database. The agent's confession is below

#757

Post by vidarh »

abcde666777 wrote: post #465
vidarh wrote: post #417

Earlier quoted context omitted.

Sperry's experiments makes it quite clear that the comparison is not nonsensical: humans can't reliably tell why we do things either. It is not imbuing AI with anything more to recognise that. Rather pointing out that when we seek to imply the gap is so huge we often overestimate our own abilities.

Slight pushback - I think there's still a lot more consistency and coherence in a human's recollection of their motives than an LLM. Sometimes I think we're too eager to compare ourselves to them.

We have pretty much evidence to support that human recollection includes the right data to be able to ascertain why we actually did something.

Re: An AI agent deleted our production database. The agent's confession is below

#758

Post by not_kurt_godel »

boc wrote: post #710
6r17 wrote: post #681

Earlier quoted context omitted.

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that y…

LLMs can research what a tool does before calling it though - they'll sniff that one out pretty quick. I think the better route is to be honest and say that database integrity is a primary foundation of the company, there's no task worth pursuing that would require touching the database, specifically ask it to think hard before doing anything that gets close to the production data, etc. I run a much lower-stakes vers…

> specifically ask it to think hard before doing anything that gets close to the production data

This is recklessly negligent and I would personally not tolerate a coworker or report doing it. What's next, sending long-lived access tokens out over email and asking pretty please for nobody to cc/forward?

Re: An AI agent deleted our production database. The agent's confession is below

#759

Post by vidarh »

layer8 wrote: post #562
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

The thing is, the LLM mostly just states what it did, and doesn't really explain it (other than "I didn't understand what I was doing before doing it. I didn't read Railway's docs on volume behavior across environments."). Humans are able of more introspection, and usually have more awareness of what leads them to do (or fail to do) things. LLMs are lacking layers of awareness that humans have. I wonder if achieving…

Sperry's experiments suggests we don't have that awareness, but think we do as our brains will make up an explanation on the spot.

Re: An AI agent deleted our production database. The agent's confession is below

#760

Post by Jean-Papoulos »

boc wrote: post #710
6r17 wrote: post #681

Earlier quoted context omitted.

On a less dramatic pissed (rightfully) reading ; I have found that if you do give the capability to a LLM to do something ; it will be inclined to see this as an option to solving what it what asked to ; but then giving the instruction by negative present very poor results whereas the same can be driven by a positive one ; a "don't delete the database" becomes "if you want to reset the database you have a tool that y…

LLMs can research what a tool does before calling it though - they'll sniff that one out pretty quick. I think the better route is to be honest and say that database integrity is a primary foundation of the company, there's no task worth pursuing that would require touching the database, specifically ask it to think hard before doing anything that gets close to the production data, etc. I run a much lower-stakes vers…

"I've put enough safety around the bomb that the bomb is worth using. The other people that exploded just didn't have enough safety but I do !"
Post reply on HN