Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

71–80 of 198 posts

Re: January 28th Incident Report

#71

Post by tonylxc »

theptip wrote: post #69
tonylxc wrote: post #51

TL;DR: "We don’t believe it is possible to fully prevent the events that resulted in a large part of our infrastructure losing power, ..." This doesn't sound very good.

The rest of the sentence is pertinent: "...but we can take steps to ensure recovery occurs in a fast and reliable manner. We can also take steps to mitigate the negative impact of these events on our users." The lessons that giants like Netflix have learned about running massive distributed applications show that you cannot avoid failure, and instead must plan for it. Now, having a single datacenter is not a good pla…

My point is: they shouldn't ONLY plan on ensuring recovery occurs fast; they should also plan on having multiple data centers, which to me is more important. It's frightening to know that such an important service is only operating in a single data center.

However, their recovery report didn't mention anything about such a plan.

<< Edited: correct a grammar error.

Re: January 28th Incident Report

#72

Post by lukeasrodgers »

tmsh wrote: post #68

> Over the past week, we have devoted significant time and effort towards understanding the nature of the cascading failure which led to GitHub being unavailable for over two hours. I don't mean to be blasphemous, but from a high level, is the performance issues with Ruby (and Rails) that necessitate close binding with Redis (i.e., lots of caching) part of the issue? It sounds like the fundamental issue is not Ruby,…

As someone with a fair bit of ruby+rails+redis experience, I don't think this is blasphemous, but I also don't think the performance issues of ruby/rails having anything to do with the failure. Generally you would cache/store something in redis not because your programming language or framework is slow, but because a query to another database is slow (or at least, slower than redis), or because redis data structures happen to be a good/quick way to store certain kinds of data.

I believe the fundamental issue was just that redis availability was taken for granted by app servers so that certain code paths/requests would fail if it wasn't available, rather than merely be slower.

Re: January 28th Incident Report

#73

Post by dice »

nemothekid wrote: post #7
bosdev wrote: post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

>Redis is really reliable (in my experience at least)

Redis has been demonstrated[0][1] to lose data under network partitions. This is particularly concerning when discussing the type of partial failure that GitHub reported.

0: https://aphyr.com/posts/283-jepsen-redis

1: https://aphyr.com/posts/307-jepsen-redis-redux

Re: January 28th Incident Report

#74

Post by theptip »

yeukhon wrote: post #15
nemothekid wrote: post #7

Earlier quoted context omitted.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

A lot of tools and services people use either don't have HA at all or don't have a native support for true distributed HA. But that can't stop people from making some HA or alike solution. I am not sure what they use Redis for but along the line of caching and key-value store they must have figured out how to invalidate data, otherwise they'd be running only a single instance of Redis. i.e. they are running "HA" just…

I was also wondering what they are using Redis for; found this article [1] from a while ago discussing Redis at Github; presumably the architecture has moved on a bit since then, but this may shed a bit of light on the subject.

[1]: https://github.com/blog/530-how-we-made-github-fast

Re: January 28th Incident Report

#75

Post by [deleted] »

yeukhon wrote: post #15
nemothekid wrote: post #7

Earlier quoted context omitted.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

A lot of tools and services people use either don't have HA at all or don't have a native support for true distributed HA. But that can't stop people from making some HA or alike solution. I am not sure what they use Redis for but along the line of caching and key-value store they must have figured out how to invalidate data, otherwise they'd be running only a single instance of Redis. i.e. they are running "HA" just…

[deleted]

Re: January 28th Incident Report

#77

Post by tonylxc »

jrockway wrote: post #56
tonylxc wrote: post #51

TL;DR: "We don’t believe it is possible to fully prevent the events that resulted in a large part of our infrastructure losing power, ..." This doesn't sound very good.

If your plan to avoid downtime is to prevent power outages, you're going to have downtime. All their sentence says is they can't prevent power outages. That's fine, because the other 1/nth of your servers are on a different power grid in a different state.

I totally share the same view that to best avoid failure is to embrace it and cope with it.

It is true that all their sentence is about recovery, however, it is disappointing that they didn't mention anything about a redundant datacenter.

Re: January 28th Incident Report

#79

Post by obituary_latte »

dice wrote: post #73
nemothekid wrote: post #7

Earlier quoted context omitted.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

>Redis is really reliable (in my experience at least) Redis has been demonstrated[0][1] to lose data under network partitions. This is particularly concerning when discussing the type of partial failure that GitHub reported. 0: https://aphyr.com/posts/283-jepsen-redis 1: https://aphyr.com/posts/307-jepsen-redis-redux

Not sure how your comment refutes the contention of reliability. Seems to me to be more a condemnation of failures that do happen (which is of course worthy of concern, but irrelevant in a conversation about stability).

Re: January 28th Incident Report

#80

Post by xzlzx »

jargonless wrote: post #43

What is this "HA" jargon? I would STFW, but searching for "HA" isn't helpful.

You could google "HA", click in the Wikipedia link that shows all the things "HA" may refer to, and deduct that the most logical thing in the list, given the context, would be this link: https://en.wikipedia.org/wiki/High_availability.
Post reply on HN