Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

81–90 of 198 posts

Re: January 28th Incident Report

#81

Post by imbriaco »

pedalpete wrote: post #13

Does Github run anything like Netflix Simbian Army against it's services? As a company by engineers for engineers with the scale that github has reached, I'm a bit surprised they are lacking a bit more redundancy. Though they may not need the uptime of netflix, an outage of more than a few minutes on github could affect businesses that rely on the service.

Google "Netflix downtime" for evidence that Netflix also has outages. Google has outages, sometimes very significant ones of Google Apps. Facebook has outages.

Complex systems fail. Period. All the time. Things like the Simian Army are fantastic tools that help you identify a host of problems and remediate them in advance, but they cannot test every combinatorial possibility in a complex distributed system.

At the end of the day, the best defense is to have skilled people who are practiced at responding to problems. GitHub has those in spades, which is why they could respond to a widespread failure of their physical layer in just over 2 hours.

The biggest win with the Simian Army isn't that it improves your redundancy. It's that it gives your people opportunities to _practice_ responses.

Re: January 28th Incident Report

#82

Post by abc_lisper »

jtblin wrote: post #29
nemothekid wrote: post #7

Earlier quoted context omitted.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

> Given the dependency in question is Redis, such a solution is probably exasperated by the fact Redis hasn't really had a decent HA solution. Redis sentinel[0] is the HA solution for redis for quite some time. [0] http://redis.io/topics/sentinel

That page says this:

Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures,

Re: January 28th Incident Report

#83

Post by eric_h »

> One of the biggest customer-facing effects of this delay was that status.github.com wasn't set to status red until 00:32am UTC, eight minutes after the site became inaccessible. We consider this to be an unacceptably long delay, and will ensure faster communication to our users in the future.

Amazon could learn a thing or two from Github in terms of understanding customer expectations.

Re: January 28th Incident Report

#84

Post by TazeTSchnitzel »

> We had inadvertently added a hard dependency on our Redis cluster being available within the boot path of our application code.

I seem to recall a recent post on here about how you shouldn't have such hard dependencies. It's good advice.

Incidentally, this type of dependency is unlikely to happen if you have a shared-nothing model (like PHP has, for instance), because in such a system each request is isolated and tries to connect on its own.

Re: January 28th Incident Report

#85

Post by beachstartup »

bosdev wrote: post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

> There's no mention of why they don't have redundant systems in more than one datacenter

sometimes reading comments on hn makes me laugh out loud.

there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post:

"Well, the reason we don't have an entire replica of our entire installation is because it costs way too much. In fact, more than double! And so far our uptime is actually 99.99% so there's no way it's worth it! You can forget about that spend! Sorry bros."

Re: January 28th Incident Report

#86

Post by dmunoz »

eric_h wrote: post #83

> One of the biggest customer-facing effects of this delay was that status.github.com wasn't set to status red until 00:32am UTC, eight minutes after the site became inaccessible. We consider this to be an unacceptably long delay, and will ensure faster communication to our users in the future. Amazon could learn a thing or two from Github in terms of understanding customer expectations.

I recently stepped into a role with a devops component, and one of my first surprises was just how slow status.aws.amazon.com was to update about ongoing issues. I had to scramble to find twitter and external forums confirmation for the client.

Re: January 28th Incident Report

#87

Post by jessaustin »

knodi123 wrote: post #55
toomuchtodo wrote: post #50

Earlier quoted context omitted.

No system is perfect; as you continue to add 9s, the cost increases steeply. Usually its just cheaper to be down for an hour or two, versus architect for the end of times.

> Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.

This seems precisely wrong. Some reading:

http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

https://www.jwz.org/doc/worse-is-better.html

[thanks for the hint 'thinkpad20! I don't know what I was thinking.]

Re: January 28th Incident Report

#88

Post by eric_h »

DarkTree wrote: post #22

I don't know enough about server infrastructure to comment on whether or not Github was adequately prepared or reacted appropriately to fix the problem. But wow it is refreshing to hear a company take full responsibility and own up to a mistake/failure and apologize for it. Like people, all companies will make mistakes and have momentary problems. It's normal. So own up to it and learn how to avoid the mistake in the…

As I said in another comment, the fact that they found an 8 minute delay from outage to status page update to be unacceptable speaks volumes to how much they value their relationship with their customers.

as an aside I feel that I'm quite fortunate to work in the EST timezone, as their outage apparently started at about 7pm my time. We have a general rule at my company to not deploy after 6pm unless an emergency fix absolutely needs to go up.

I saw the title of the story and said to myself, what outage? :P

Re: January 28th Incident Report

#89

Post by atom_enger »

dmunoz wrote: post #86
eric_h wrote: post #83

> One of the biggest customer-facing effects of this delay was that status.github.com wasn't set to status red until 00:32am UTC, eight minutes after the site became inaccessible. We consider this to be an unacceptably long delay, and will ensure faster communication to our users in the future. Amazon could learn a thing or two from Github in terms of understanding customer expectations.

I recently stepped into a role with a devops component, and one of my first surprises was just how slow status.aws.amazon.com was to update about ongoing issues. I had to scramble to find twitter and external forums confirmation for the client.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues.

I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It's a sad state of affairs when you have to do this and Amazon doesn't seem interested in fixing it.

Re: January 28th Incident Report

#90

Post by dice »

obituary_latte wrote: post #79
dice wrote: post #73

Earlier quoted context omitted.

>Redis is really reliable (in my experience at least) Redis has been demonstrated[0][1] to lose data under network partitions. This is particularly concerning when discussing the type of partial failure that GitHub reported. 0: https://aphyr.com/posts/283-jepsen-redis 1: https://aphyr.com/posts/307-jepsen-redis-redux

Not sure how your comment refutes the contention of reliability. Seems to me to be more a condemnation of failures that do happen (which is of course worthy of concern, but irrelevant in a conversation about stability).

I am using "reliability" in the sense of RAS[0]. An HA datastore which erroneously ACK's writes has lowered reliability, as there are known cases where it gives incorrect outputs.

0: https://en.wikipedia.org/wiki/Reliability,_availability_and_...

Post reply on HN