It must be nice to know that the majority of your customers are familiar enough with the nature of your work that they'll actually understand a relatively complex issue like this. Almost by definition, we've all been there.
January 28th Incident Report
101–110 of 198 posts
Re: January 28th Incident Report
#102Does Github run anything like Netflix Simbian Army against it's services? As a company by engineers for engineers with the scale that github has reached, I'm a bit surprised they are lacking a bit more redundancy. Though they may not need the uptime of netflix, an outage of more than a few minutes on github could affect businesses that rely on the service.
Google "Netflix downtime" for evidence that Netflix also has outages. Google has outages, sometimes very significant ones of Google Apps. Facebook has outages. Complex systems fail. Period. All the time. Things like the Simian Army are fantastic tools that help you identify a host of problems and remediate them in advance, but they cannot test every combinatorial possibility in a complex distributed system. At the en…
More than practicing responses, Chaos Monkey and Failure Injection Testing allow us to verify that we don't have unexpected hard dependencies. Sometimes you find out that your service can't start if another one becomes latent, in which case you can plan for it by adding redundancy/extra capacity, fallbacks or working in degraded mode.
Re: January 28th Incident Report
#103Earlier quoted context omitted.
What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…
If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
[deleted]
Re: January 28th Incident Report
#104Earlier quoted context omitted.
Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.
>Redis is really reliable (in my experience at least) Redis has been demonstrated[0][1] to lose data under network partitions. This is particularly concerning when discussing the type of partial failure that GitHub reported. 0: https://aphyr.com/posts/283-jepsen-redis 1: https://aphyr.com/posts/307-jepsen-redis-redux
I meant Redis is really reliable as a single instance. If you reread my post I mentioned that Redis doesn't have a decent HA solution.
Re: January 28th Incident Report
#105Weird that they didn't say what caused the power outage and what the mitigations are for that.
I'm also confused about how the racks would lose power. Surely they had UPSes.
UPSs don't always cover everything. There are systems that are considered critical that are on UPS, and others that are considered restartable that might not be. There are a lot of tradeoffs in a data center. Having full UPS and generator backup capacity for everything gets very expensive.
Re: January 28th Incident Report
#106Earlier quoted context omitted.
> Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.
This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]
Just a note: if you don't indent your links they'll be made clickable by the markup engine, which is convenient in general and especially for those of us on smart phones. :)
Re: January 28th Incident Report
#107Earlier quoted context omitted.
Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.
> Given the dependency in question is Redis, such a solution is probably exasperated by the fact Redis hasn't really had a decent HA solution. Redis sentinel[0] is the HA solution for redis for quite some time. [0] http://redis.io/topics/sentinel
I haven't heard good things about Redis Sentinel nor am I sure of their failure modes, which is why I wouldn't describe it as decent.
I haven't kept up to date on "Sentinel 2" that was launched with 3.0 so the situation might have changed.
Re: January 28th Incident Report
#108What is this "HA" jargon? I would STFW, but searching for "HA" isn't helpful.
You could google "HA", click in the Wikipedia link that shows all the things "HA" may refer to, and deduct that the most logical thing in the list, given the context, would be this link: https://en.wikipedia.org/wiki/High_availability .
Would it have been so hard to just type "high availability" rather than making him feel bad for being one of today's 10,000? https://xkcd.com/1053/
Re: January 28th Incident Report
#109What is this "HA" jargon? I would STFW, but searching for "HA" isn't helpful.
I had to STFW to figure out what STFW meant...TIL.
Re: January 28th Incident Report
#110> Remote access console screenshots from the failed hardware showed boot failures because the physical drives were no longer recognized.
I'm getting flashbacks. All of the servers in the DC reboot and NONE of them come online. No network or anything. Even remotely rebooting them again we had nothing. Finally getting a screen (which is a pain in itself) we saw they were all stuck on a grub screen. Grub detected an error and decided not to boot automatically. Needless to say we patched grubbed and removed this "feature" promptly!