We deleted the production database by accident • Hacker News Board

Re: We deleted the production database by accident

#341

Post by qz2 » Sun, Oct 18, 2020, 11:32 AM UTC

ClumsyPilot wrote: post #319

qz2 wrote: post #259

Earlier quoted context omitted.

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

"fear required when working out what decision to make" People like you keep making the same mistake, creating companies/organisations/industries/societies that run on fear of failure. We've tried it a thousand times, and it never works. You can't solve crime by making all punishments hearsh death, we've tried that in 1700 in Britain and crimerate was sky high. This culture gave us disasters in USSR and famine in Chin…

I think my point is probably being misunderstood and that is my fault for explaining it poorly. See I fucked up :)

The fear I speak of is a personal barrier which is lacking in a lot of people. They can sleep quite happily at night knowing they did a shitty job and it's going to explode down the line. It's not their problem. They don't care.

I can't do that. Even if there are no direct consequences for me.

This is not because I am threatened but because I have some personal and professionals standards.

Re: We deleted the production database by accident

#342

Post by qz2 » Sun, Oct 18, 2020, 11:33 AM UTC

rawoke083600 wrote: post #337

qz2 wrote: post #331

Earlier quoted context omitted.

The point is that if you take responsibility then you're taking pride in your work, are invested in it and willing to invest in self-improvement and introspection rather than doing the minimum to get to a minimum viable solution. The outcome of this is an increase in quality and a decrease in risk.

Wow - "...invest in self-improvement and introspection.." I would hate for that to be our system reliability improvement methodology. Ok fine now I'm being slightly a "d k" but really ?

Well you can also apply a QMS if you want but all that does is generate paperwork full of accepted risks...

Re: We deleted the production database by accident

#343

Post by azeirah » Sun, Oct 18, 2020, 11:35 AM UTC

oppositelock wrote: post #151

You have to put a lot of thought into protecting and backing up production databases, and backups are not good enough without regular testing of recovery. I have been running Postgres in production supporting $millions in business for years. Here's how it's set up. These days I use RDS in AWS, but the same is doable anywhere. First, the primary server is configured to send write ahead logs (WAL) to a secondary server…

Quick tip for anyone learning from this thread.

If you're using MySQL, it's called a binary log and not a Write Ahead Log, it was very difficult to find meaningful Google results for "MySQL WAL"

Re: We deleted the production database by accident

#344

Post by qz2 » Sun, Oct 18, 2020, 11:35 AM UTC

rawoke083600 wrote: post #325

qz2 wrote: post #302

Earlier quoted context omitted.

No. There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another. I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome. If I fuck up I’m the first person to put my hand up. C…

I think you missing point, I love the idea about "bringing aviation methodology" to lower error/f-up-rates" for the software industry. No one is not saying don't take responsibility, they are saying - as I understood it: Have a "systematic-approach" to the problem, the current system for preventing "drunk pilots or the wiping of production db's are not sufficient" - improve the system ! ! All the "taking responsibili…

Yes. CTO-2 is my approach. As the CTO I fucked up because I didn't have that process in place to start with. To buck stops at me.

CTO-2 also has the responsibility of making sure that everyone is educated on this issue and can communicate those issues and worries (fears) to his/her level effectively because prevention is better than cure. Which is my other point.

Re: We deleted the production database by accident

#345

Post by li4ick » Sun, Oct 18, 2020, 11:44 AM UTC

junglejoose wrote: post #32

You aren’t a real engineer until you do this. So congrats on the promotion! :)

Yeah, imagine if a bridge engineer said the same thing: "You aren't an engineer until your bridge collapses. Congrats!" I am starting to hate tech culture. Nobody cares about correctness and discipline. Mention "math" and everybody spreads like cockroaches.

Re: We deleted the production database by accident

#346

Post by jbverschoor » Sun, Oct 18, 2020, 11:44 AM UTC

> Why? This is something we’re still trying to figure

Probably the admin has set the hba config to trust localhost. Solution - don't use the same db name in prod jsut to be sure

Re: We deleted the production database by accident

#347

Post by adwn » Sun, Oct 18, 2020, 11:45 AM UTC

tamrix wrote: post #225

Nextgrid wrote: post #89

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

I think we're hitting peak tech. All this "technical" knowledge just dates itself in a year's time anyway. Eventually, you come to realise that the more tech you've got, the more problems you have. . Now developers spend more time googling errors and plugging in libraries and webservices together than writing any actual code. Sometimes I wish for a techless cloudless revolution when we just go back to the foundations…

> Sometimes I wish for a techless cloudless revolution when we just go back to the foundations of computers and is use plain text wherever possible.

... because software in the 60s/70s/80s was so reliable and bug-free?!

Re: We deleted the production database by accident

#348

Post by pvorb » Sun, Oct 18, 2020, 11:46 AM UTC

junon wrote: post #230

pvorb wrote: post #209

Earlier quoted context omitted.

Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.

That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively. Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.

Why do you think they are hacks? Could you please elaborate?

In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.

Re: We deleted the production database by accident

#349

Post by ojnabieoot » Sun, Oct 18, 2020, 11:48 AM UTC

rocqua wrote: post #257

vanviegen wrote: post #211

Earlier quoted context omitted.

> No one owned up to it, but had a pretty good idea who it was. That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.

Misclicking is a tiny forgivable mistake. Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

It sounds like whoever did it might not even be aware they were responsible:

> in the dropdown menu of the MongoDB browser, exit & drop database were next to each other

So maybe they signed off for the night without realizing anything was wrong.

Re: We deleted the production database by accident

#350

Post by benhurmarcel » Sun, Oct 18, 2020, 11:52 AM UTC

ClumsyPilot wrote: post #332

vox17 wrote: post #315

This line's the winner for me : "Thankfully nobody’s job is at risk due to this disaster. The founder is not going to fire the developer – because they are one and the same person."

To be fair he could, and least in theory - he could get someone else to do development, for money or equity, for the project and do something else himself.

It's just one guy working in the evening on a side project. There's no revenue, and it runs on $171/month [1]. He's not going to hire anyone.

[1] https://keepthescore.co/blog/posts/monetizing-keepthescore/