Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

231–240 of 269 posts

Re: Postmortem of database outage of January 31

#231

Post by user5994461 »

pzh wrote: post #111
pfarnsworth wrote: post #102

Earlier quoted context omitted.

At a startup I worked at, the VP of Engineering was brilliant. He was probably the smartest person I've ever worked for, and the most hard working. He was online almost all hours of the day, working. He also insisted on a 9-to-5 schedule for all the engineers, because he believed that killing your engineers with work was not a scalable way to build a team. He was great. But the first month I was there, I kept pressin…

Maybe in his own cocky way, his disaster recovery plan was to be smart enough to never have a disaster. The problem with this approach is that it only works for a small company, and many people figure out the hard way that smarts don't scale.

Or maybe it was a small startup where him and the poster were the only tech guys and they struggled to performs all the tasks to be done, not limited to backup and DR.

Re: Postmortem of database outage of January 31

#232

Post by lomnakkus »

fredsir wrote: post #212
lomnakkus wrote: post #206

Earlier quoted context omitted.

> They should have spun up a new server to act as secondary the moment replication failed. In a perfect world everything is cluster-ready &c at the outset. In this world it usually... isn't. EDIT: ... and I'd posit that such cluster-readiness actually isn't worth it most of the time.

You don't think it's worth it most the time because of the hassle of setting up and managing a cluster, or because clusters in and of it self is not necessary for most?

The latter. I think it's a 1% business case, basically. I mean, if we can get 80% benefit without excessive cost, then it's obviously a good idea. (And I do use Ansible/Docker and the like, but it's not entirely without friction... which is where the cost/benefit analysis comes in.)

EDIT: Obviously, if you really need clustering, then you need it, but IME people tend to overestimate their needs drastically. Everybody wants to be Big Data, but almost nobody actually is.

Re: Postmortem of database outage of January 31

#233

Post by user5994461 »

bogomipz wrote: post #90
elliotec wrote: post #85

Earlier quoted context omitted.

They also openly pay well below market rate, so they should expect to get what they pay for.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Well, it's hard to judge the algorithm. Some numbers are decent, if you are given "lead with lots of experience".

1) It's hard to tell how they assign rank. If they accept any programmer that shows up, the rates are fine. If they have Google level interviews to filter for only Google level candidates, who will join at "junior" level, it's terrible.

2) The numbers are in dollars, thus they are utterly meaningless. It's not a job, it's gambling with the exchange rate and the exchange fees.

Re: Postmortem of database outage of January 31

#234

Post by user5994461 »

cookiecaper wrote: post #167
milesrout wrote: post #161

Earlier quoted context omitted.

I can't tell if this is some kind of joke or not. There's absolutely nothing wrong, whatsoever, with having a public IP address on a production database server.

I wouldn't say there's "absolutely nothing wrong with it". Someone may have a valid use case to leave a database server exposed to the public internet, but they probably don't. The only things that should be public facing are things that clients need to access directly. In most cases, that's just an HTTP server. In GitLab's case, it's an HTTP server and a git server. One of the most important principles of infrastruc…

Little correction: HTTP servers should not be publicly accessible. Only a couple of load balancers need to be accessible.

Re: Postmortem of database outage of January 31

#235

Post by bshanks »

cookiecaper wrote: post #209
YorickPeterse wrote: post #186

Earlier quoted context omitted.

> 1. notifications go through regular email. Email should be only one > channel used to dispatch notifications of infrastructure events. Tools > like VictorOps or PagerDuty should be employed as notification > brokers/coordinators and notifications should go to email, team chat, and > phone/SMS if severity warrants, and have an attached escalation policy so > that it doesn't all hinge on one guy's phone not being dea…

> Hot standby is also supported just fine out of the box, you just need something third party for the actual load balancing. I'm aware that hot standby is supported, though it's not the default configuration for the standby server (default and safest is a standby mode that you can't query at all; hot standby introduces possible conflicts between hot read queries and write transactions coming in from the WAL, so if fa…

Regarding the spam user deletions, I am not involved but my interpretation of what GitLab wrote ( https://gitlab.com/gitlab-org/gitlab-ce/issues/27581 https://gitlab.com/MrChrisW/gitlab-anti-spam-toolkit/issues/... ) is that, currently, there is no auto-delete running, and a human must approve each deletion.

Re: Postmortem of database outage of January 31

#236

Post by bbcbasic »

was_boring wrote: post #146
bbcbasic wrote: post #136

Earlier quoted context omitted.

Supply and demand

Of what? A developer in that location? What is that benefit? Having worked on both coasts, I can say that the quality is the same. Culture and quantity appears to be the biggest difference.

Yep a developer in that location. The market puts a high price on that. Regardless of same quality people being cheaper elsewhere. If you think the market is wrong then go try to arbitrage it. Maybe you can...

Re: Postmortem of database outage of January 31

#237

Post by lojack »

viraptor wrote: post #187
cookiecaper wrote: post #160

Earlier quoted context omitted.

The red PS1 would've clearly indicated to the engineer that he was typing `rm -rf ...` on the _master_, not the secondary. This assumes that the master and secondary would have differing prompts based on their relative importance.

That would help, but that's not what OP advocated. Sure, you can improve on those ideas. I was mainly pointing out that saying "it's unlikely to happen to me" was a bit dangerous and too sure, if most of the reasons do not apply to the situation.

Would the steps I describe prevent actions taken in the GitLab incident? I would never make no assumptions to that. Maybe. Maybe not. Did I say following those steps would make it unlikely to happen to you? No. That's why I prefaced it with "I'm not a sysadmin." Would it prevent cases described by the person I was responding to? Absolutely. Not 100% of the time, but some percentage of the time.

So, I'll say it more clearly, and you can mark my words. It's unlikely I'll ever log into a production system, type the wrong command, and do something bad as a result.

Could I deploy code that does very bad things to production? Yes. It'll probably happen to me. Is that the situation described above? No.

I treat logging into a production system as if one wrong move could result in me losing my job. Why? Because one wrong move could result in me losing my job. I'm not joking when I say I avoid logging into a production system like the plague. It's unlikely to happen to me because its extremely rare for me to put myself in a situation where I could let this happen. There's almost always better alternatives that I'll resort to, well before doing anything like this.

Re: Postmortem of database outage of January 31

#238

Post by lojack »

viraptor wrote: post #148
lojack wrote: post #124

Earlier quoted context omitted.

I'm no sysadmin, and I know mistakes are inevitable and all... but I find this kind of mistake is unlikely to come from me. I feel as though a lot of developers are too nonchalant about production boxes. I think one or two close calls where I nearly did this exact thing served as a good wakeup call for me. Steps I personally take to avoid this: - Avoid prod boxes like the plague - Set up a prompt (globally) to make i…

I don't think most of your points really apply though. They were setting up replication in production, so they had to work on production boxes. Setting prompt to say just "production" wouldn't help for the same reason. Production was intended. Peer review though - yes. That could help. I wouldn't say "I'm unlikely to make that mistake" - it's likely to go on the famous last words list...

Sure, maybe the point about "PRODUCTION" may not have applied, I was mainly commenting on OPs post about how we've all been in a situation and the steps I take to avoid such situations. I'm curious about your arguments about the remaining 3/4 of the steps I take and how those wouldn't have applied to this situation?

Re: Postmortem of database outage of January 31

#239

Post by AlexCoventry »

Thank you for this informative postmortem and mitigation outline.

Are any organizational changes planned in response to the development friction which led to the outage? It seems to have arisen from long-standing operational issues, and an analysis of how prior attempts to address those issues got bogged down would be very interesting.

Re: Postmortem of database outage of January 31

#240

Post by jjirsa »

matt_wulfeck wrote: post #157
KayEss wrote: post #132

The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good…

This type of thing always sounds good and all, but the reality is people get desperate and emotional when their website is down and everyone wants it up ASAP.

Desperate and emotional is no way to run a business
Post reply on HN