Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

251–260 of 456 posts

Re: We deleted the production database by accident

#251

Post by robryan »

dingdingdang wrote: post #231
toredash wrote: post #201

Earlier quoted context omitted.

2 cents his hosts file points localhost to the prod db IP

Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") tha…

Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.

Re: We deleted the production database by accident

#252

Post by drdaeman »

kjaftaedi wrote: post #216
toredash wrote: post #201

Earlier quoted context omitted.

2 cents his hosts file points localhost to the prod db IP

I think so as well. It doesn't even make sense to connect to a managed database using 'localhost'. Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.

That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`).

It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)

Re: We deleted the production database by accident

#253

Post by ozorOzora »

greggman3 wrote: post #164
Nextgrid wrote: post #89

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

What? I thought that was just the opposite. The advantage of serverless is that I pay AWS to make backups so I don't have to. I mean under time pressure if it do it myself I'll skip making backups, setting permissions perfectly, and making sure I can actually restore those backups. If I go with a microservice, the whole point is they already do those things for me. No?

I'm pretty sure the advantage of serverless is that you can use microservices for your MVPs. The gains for a one-man team might not be ovious, but I like to believe that once a project needs scaling it is not as painful as tearing down the monolith.

Re: We deleted the production database by accident

#254

Post by dmitriid »

t0mas88 wrote: post #250
suzakus wrote: post #246

Earlier quoted context omitted.

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

It isn't about the importance of this one database, it's about the cultural issue in most of the sector that the parent comment was pointing out: we far too often blame the user/operator calling them stupid, while every human makes mistake, it's inevitable.

The Russian joke about investigating is "Punish the innocent, award the uninvolved"

Re: We deleted the production database by accident

#255

Post by qz2 »

suzakus wrote: post #246
t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

This time.

Some days it’s just an on line community that gets burned to the ground.

Other days it’s just a service tied into hundred of small businesses that gets burned to the ground.

Other says it’s massive financial platform getting burned to the ground.

I’m responsible for the latter but the former two have had a much larger impact for many people when they occur. Trivialising the lax administrative discipline because a product isn’t deemed important is a slippery slope.

We need to start building competence in to what we do regardless of what it is rather than run on apologies because it’s cheaper.

Re: We deleted the production database by accident

#257

Post by rocqua »

vanviegen wrote: post #211
fma wrote: post #117

Earlier quoted context omitted.

Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog. No one owned up to it, but had a pretty good idea who it was.

> No one owned up to it, but had a pretty good idea who it was. That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.

Misclicking is a tiny forgivable mistake.

Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

Re: We deleted the production database by accident

#258

Post by madsbuch »

Things like these happen, and we should be compassionate towards them.

Often small changes to the structure drastically reduce probability of stuff like this happening.

Eg. we use docker to setup test and dev databases and seed from (processed) dumps. When we need to clean our database, we simply put down the docker container. Ie. we do not need to implement destructive database cleanup eliminating structure that could potentially fail.

Having policies about not accessing production database directly (and allow the extra time for building tooling around that policy), good preview / staging environments, etc. All fail eliminating structure.

Re: We deleted the production database by accident

#259

Post by qz2 »

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I disagree.

Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make.

What happens is we have growing complacency and disassociation from consequences.

Do you press the button on something potentially destructive because your are confident it is ok through analysis, good design and testing or confidence it is ok through trite complacency?

The industry is mostly the latter and it has to stop. And the first thing is calling bad processes, bad software and stupidity out for what it is.

Honestly these guys did good but most will try and hide this sort of fuck up or explain it away with weasel words.

Re: We deleted the production database by accident

#260

Post by danjac »

cranekam wrote: post #177
danellis wrote: post #77

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

The whole piece has a slightly annoying flippant tone to it. We were drunk! Computers just.. do this stuff sometimes! Better to sound contrite and boring in such a situation IMO. Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.

The Exxon Valdez comes to mind - the company blamed the drunk captain, but this was just part of huge systemic failures and negligence.
Post reply on HN