21–30 of 50 posts
GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.
have you tried HackPad instead of docs?
One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.
One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.
Very true, we'll update the graph server that was misconfigured.
One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.
If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
Earlier quoted context omitted.
have you tried HackPad instead of docs?
I have yet to understand why HN:ers like HackPad. It is at best almost as good as GDoc:s on some of the features.
One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.
Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
Earlier quoted context omitted.
Very true, we'll update the graph server that was misconfigured.
This is a very open and candid write-up. Very much appreciated by the community. Too many companies try to hide/cover-up their outages, but Gitlab is letting it all hang out, their mess-ups as well as things outside their control. I think that speaks a lot to the character of the company, and shows they really care.
We're talking a break until 16:00 CEST, many questions are answered already and TODO's are given after the => in the document.
Earlier quoted context omitted.
Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
This starts out to be the way most companies think, then before you know it you are big enough to start opening datacenters in other countries, then you have to deal with timestamp conversion.