Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

361–370 of 621 posts

Re: GitLab Database Incident – Live Report

#361

Post by sytse »

onetom wrote: post #264

Thank you GitLab people for writing a log! It's just as important as the service you are providing. Keep up the good job and never be afraid to talk about your mistakes. It would be also very educational if you could try to do a "5 whys" session and share it too. The person who made the mistake deserves a bit of rest, that's for sure. I wish she or he is supported by the team emotionally and not just being blamed.

Totally agree with doing the 5 why's, see https://news.ycombinator.com/item?id=13539595

Re: GitLab Database Incident – Live Report

#362

Post by jfindley »

dijit wrote: post #353
jfindley wrote: post #350

"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented" This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for thi…

I don't share your observation at all. If we're talking normal replication then I can tell you for certain that the built in replication system is completely rock solid (especially compared to MySQL). If you're talking multi-master replication, then Citus is pretty solid, but not nearly as solid as replication. If you're talking statement based replication, well, it's the same as all databases. Here be dragons.

The built-in replication may be good - but it's pretty new. Replication slots, which are IMO vital for making the built-in replication non-fragile, only arrived in 9.4, which is pretty recent of a release. I wonder how widely tested it is, given that? How many people are actually using it for large workloads?

Citrus I hadn't heard of though - that's interesting, thanks.

Re: GitLab Database Incident – Live Report

#363

Post by yxhuvud »

debaserab2 wrote: post #258
iagooar wrote: post #245

Earlier quoted context omitted.

I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.

You can check out their codebase here: https://github.com/gitlabhq/gitlabhq Just looking at their gemfile is rather telling: a couple hundred gems. I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve. They're probably at the point where they really want to think about splitting off of their monolith codebase and into microservices.

It is fairly large, but I still find it more organized than some examples I've seen.

Also, I don't see another very common issue with big gemfiles in that they don't seem to have multiple solutions of one thing in there (ie multiple REST clients, DB mockers, etc).

Re: GitLab Database Incident – Live Report

#364

Post by on_and_off »

late2part wrote: post #217
schleppy_oc wrote: post #208

Earlier quoted context omitted.

The best Ops people I have worked with (looking at you Dennis and Alan) repeat everything back that I say. More than once I have caught mistakes in my approach simply by hearing someone else repeat back exactly what I just said.

The best ops people have made all the mistakes they're helping you avoid.

I think this applies to pretty much any field : http://thecodelesscode.com/case/100

Re: GitLab Database Incident – Live Report

#366

Post by corecoder »

nodesocket wrote: post #311
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

If using iterm you can set a "badge" text on a terminal window that shows up as an overlay. Super useful when you have lots of SSH sessions open to different servers.

iterm badges have saved me many times.

Re: GitLab Database Incident – Live Report

#368

Post by sytse »

grhmc wrote: post #102
sytse wrote: post #42

Earlier quoted context omitted.

That is why we went with initials. And I hope customers understand the blame is with all of us, starting with me. Not with the person in the arena. https://twitter.com/sytses/status/826598260831842308

It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.

We are all responsible for this, starting with me. We don't blame any person. References to people are just to determine what happend. Also see https://twitter.com/sytses/status/826598260831842308

Re: GitLab Database Incident – Live Report

#369

Post by sytse »

mrmondo wrote: post #149
sytse wrote: post #55

Earlier quoted context omitted.

We've hired some great new people recently but as you can see there is still a lot work to do. https://about.gitlab.com/jobs/production-engineer/

I've just DM'd you in twitter with some PostgreSQL advice ;)

Thanks for that, we'll collaborate on it at the bottom of https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx...

Re: GitLab Database Incident – Live Report

#370

Post by sytse »

knocte wrote: post #259
sytse wrote: post #48

Earlier quoted context omitted.

Thanks, not feeling great about the job we're doing today, but we'll learn from this.

Losing 6 hours of data is not a big deal! Kudos!

It is a big deal to us. It should never be more than an hour and we should be able to recover much quicker. But thanks for the kudos.
Post reply on HN