Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab.com Downtime Postmortem

docs.google.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: GitLab.com Downtime Postmortem

#41

Post by sciurus »

"6. Production filesystem IS NOT mounted at this point

9. gitlab-ctl start starts the GitLab with the staging data in /var/opt/gitlab on the root filesystem that doesn’t have any production data. At this point logs report that the production db doesn’t exist which is correct because we are not on the production file system. No production data has been touched at this point. The Gitlab web UI is not responding (502 error from nginx)"

This is what strikes me as needing addressing. There shouldn't be staging data that is normally hidden by mounting the production filesystem. If the production database isn't there, Postgres should fail to start. The Postgres team is pretty adamant that otherwise bad things can happen; see http://www.postgresql.org/message-id/12168.1312921709@sss.pg...

Re: GitLab.com Downtime Postmortem

#42

Post by res0nat0r »

wpietri wrote: post #31

The bit about the large repositories reminds me tangentially of something I read about Amazon: when tuning for performance, they don't look at, say, the median response time. They look at the 99.9% level. If I recall rightly, their example was a customer searching their old orders. The customers with the slowest response times were their very best customers, and they wanted those people to have at least as good as an…

Yup, AWS uses this metric everywhere (TP99). If 1% of your requests take 1000% more time than everything else, then you need to optimize and ensure this doesn't screw everyone.

Here is a bit more info: http://codesith.blogspot.com/2012/06/tp99.html

Re: GitLab.com Downtime Postmortem

#43

Post by wpietri »

scott_karana wrote: post #38
wpietri wrote: post #26

Earlier quoted context omitted.

The reason I've seen it used is that it's open source and therefore can be hosted by the entity involved.

It is?! Where's the source code? Their GitHub page is rather sparse, and the only hint I've found is this tweet: https://twitter.com/hackpad/status/407651583598813184

Sorry, I'm thinking of Etherpad:

http://etherpad.org/

The source code of which is here:

https://github.com/ether

Hackpad is apparently a fork of that:

https://hackpad.com/Whats-the-difference-between-Hackpad-and...

Re: GitLab.com Downtime Postmortem

#44

Post by scott_karana »

wpietri wrote: post #43
scott_karana wrote: post #38

Earlier quoted context omitted.

It is?! Where's the source code? Their GitHub page is rather sparse, and the only hint I've found is this tweet: https://twitter.com/hackpad/status/407651583598813184

Sorry, I'm thinking of Etherpad: http://etherpad.org/ The source code of which is here: https://github.com/ether Hackpad is apparently a fork of that: https://hackpad.com/Whats-the-difference-between-Hackpad-and...

Cool, thanks!

Re: GitLab.com Downtime Postmortem

#45

Post by sytse »

sciurus wrote: post #41

"6. Production filesystem IS NOT mounted at this point 9. gitlab-ctl start starts the GitLab with the staging data in /var/opt/gitlab on the root filesystem that doesn’t have any production data. At this point logs report that the production db doesn’t exist which is correct because we are not on the production file system. No production data has been touched at this point. The Gitlab web UI is not responding (502 er…

We fixed part of the root issue by restarting GitLab automatically when we configure a server as master, preventing it from continuing with the staging data. The staging data was created automatically by gitlab-ctl. I agree that this is confusing and created the item: "Why did GitLab start when it shouldn't?" in the postmortem. Thanks for raising this point.

Re: GitLab.com Downtime Postmortem

#46

Post by sytse »

jbk wrote: post #37

Well, this is not back for us: our repositories are corrupted...

I'm sorry to hear you are experiencing problems. This is the first report we see of possible corruption. Please contact support@gitlab.com and note the urls and commands that do not work for you. Edit: there another report I just found: https://gitlab.com/gitlab-com/support-forum/issues/2

Re: GitLab.com Downtime Postmortem

#47

Post by supine »

peterwwillis wrote: post #39
felixgallo wrote: post #22

One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.

Of course all hardware clocks should be set UTC, but there's a lot of benefit from using the locale to understand what's going on in relation to the human viewing the log. If i'm a human waking up at 3AM and I look at the logs for this server, the first thing i'm going to think is "is the time on this server the same as where I am?" followed by "how long ago did these events occur in relation to me?" The easiest way…

I think you just made a better case for having everything in UTC. Instead of having to remember multiple offsets, one for each location, you just need to know your current offset to get a sense of relativity (i.e. "ok, that occurred an hour ago") and the correlation across different locations and sources takes care of itself.

I'm not saying that a log aggregator is not needed, they are still an important of any system as your 3rd paragraph clearly explains, but your 2nd paragraph actually makes the case for keeping everything in UTC regardless.

Re: GitLab.com Downtime Postmortem

#48

Post by supine »

felixgallo wrote: post #35
wpietri wrote: post #25

Earlier quoted context omitted.

Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.

That's a good question. UTC is almost certainly still the best candidate even in that case because: * it's not affected by daylight savings time, which in many candidate timezones causes at least two complex and error fraught time discontinuities every year, and which requires manual intervention at the whims of the US Congress in the USA. * UTC is the lingua franca of machine time communication -- if you ever have t…

...and just when you thought it was safe to go back outside leap seconds rear their ugly head and ruin your day.

Re: GitLab.com Downtime Postmortem

#49

Post by supine »

forgottenpass wrote: post #34
maddev wrote: post #30

Earlier quoted context omitted.

Why would you ever need to convert a timestamp? You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). The returned time is obviously +0000 == UTC. If it's not your language/os/software sucks and its no wonder your hair is on fire.

You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). There's your conversion. I'm not saying it's not easy, I'm saying you're just pulling you out of a hole that your tools were configured to dig.

An example of some of the pitfalls involved: http://search.cpan.org/~drolsky/DateTime-1.10/lib/DateTime.p...

Re: GitLab.com Downtime Postmortem

#50

Post by maddev »

korzun wrote: post #40
maddev wrote: post #30

Earlier quoted context omitted.

Why would you ever need to convert a timestamp? You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). The returned time is obviously +0000 == UTC. If it's not your language/os/software sucks and its no wonder your hair is on fire.

Because it's always easier said than done. Multiple applications in different languages that are running on different environments and managed by different teams posses a real challenge.

It's not like am speculating. I implemented these things. The hardest part is agreeing on TZ symbols (e.g. Indian Standard Time vs Ihoa Standard Time). E.g. its not very hard to get right.
Post reply on HN