111–120 of 198 posts
> Because we have experience mitigating DDoS attacks, our response procedure is now habit and we are pleased we could act quickly and confidently without distracting other efforts to resolve the incident. The thing that fixed the last problem doesn't always fix the current problem.
Weird that they didn't say what caused the power outage and what the mitigations are for that.
I'm also confused about how the racks would lose power. Surely they had UPSes.
Once it involved fire alarms, which trigger safety shutdowns within a suite. The other involved a failed static switch panel - ie, the things that aren't mean to be able to fail.
That would have given them immediate co text and not wasting time on DDOS protection
Earlier quoted context omitted.
What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…
If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
1. File ticket.
2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour).
3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalling tactic.
4. Play the dumb question/obvious response dance, waiting an hour or more for a response each time.
5. If you are lucky (usually a couple hours in now) they acknowledge there's some problem (but never give you any detail) and escalate your ticket to a higher level internal team. If you are unlucky, you are calling up your account rep (do you even have one??) and getting them to harass tech support.
6. Usually around now the problem "magically" disappears if you haven't already fixed it yourself.
7. If you are lucky, a few hours, days, weeks later you get a response asking if you are still having the problem? You, of course, are NOT having the problem since you long ago solved it yourself. If you are really unlucky they try to schedule a meeting with one of their "solution architects" who is then going to waste an hour of your time telling you how to properly "design" your software for the cloud (i.e. trying to sell you on even more of their services).
8. Ticket is closed having never gotten to the bottom of the problem, maybe get a survey.
I've never seen this go down differently. Filing more tickets isn't going to change this. You want to really change things?
STOP PAYING THEM!
If a few mid-sized customers stop paying them and make a big-stink when they do it, then I guarantee you things will change! Until then, they have little incentive to improve and the big customers have a direct line to Amazon so they can circumvent all this crap. It's up to the small and mid-sized customers to push for change and the most effective way to do this would be to spend your money elsewhere.
This just shows how difficult it is to avoid hidden dependencies without a complete, cleanly isolated, testing environment of sufficient scale to replicate production operations and do strange system fault scenarios somewhere that won't kill production.
Or use the Netflix model: Chaos testing in production.
Earlier quoted context omitted.
If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
I take it you have no experience filing tickets with them. A typical ticket goes something like this: 1. File ticket. 2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour). 3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalli…
Earlier quoted context omitted.
Is there a timeline to how long it took them to figure out Redis was down? Because having experienced the same, you get an alert. Cool. HA-Proxy says app servers are down. Ok. You SSH in and see that everything looks ok but the processes are bouncing. You tail the logs to find out why (obviously lots of these steps could be optimized). Within a few seconds you spot the error connecting to Redis. A minute later you've…
Is it really "shameful"? Running systems like this at scale is hard. We're not talking about redundant power systems for an ICU Ward in a hospital. We're talking about a website which powers a sliver of the first world. You bet they busted their ass to get this fixed and shared their learnings with us. I'm extremely grateful for this and yeah it inconvenienced my morning but nothing more. You make it sound so easy. I…
I learned they had an unfortunate power outage. Then it took about two hours to determine that the Redis servers weren't booting, and to failover.
That's honestly pretty unimpressive no matter how you slice it.
Setting up a Redis server is easy. Sharding it is easy. Setting up slave systems is easy.
What's hard is, like most things in life and tech, planning. Planning for failure. Practicing failure. Not by insisting you need a monkey-army, which sounds cool and fun, but by having a staging environment which is mundane and boring. Pull the power plug on a server. See how long it takes to identify the culprit and recover. Figure out what you could have done to identify the issue quicker. Figure out how you could recover quicker.
You're running Redis, and you haven't even setup slave systems in different cabinets? You've never tried pulling the plug on the JBOD to see what might happen? These servers apparently weren't even pinging. Why did it take more than 10 seconds to find out they weren't running? Why wasn't there a dashboard for basic system/process status across all services? Is GitHub's operations budget really that tight?
Setting up a pair of OpenBSD boxes with pf and HA-Proxy is easy. Making sure CARP picks up on the failover system when the primary fails is pretty easy. Scheduling it, testing it so you know it's actually going to work when you need it to is the hard part. Holding people accountable for it's uptime when the log says that hasn't been done this month is the hard part.
Setting up some FreeBSD boxes with DAS is easy. Giving each shard it's own slaves is easy. Monitoring the slaves to make sure they're actually useful is hard. Making sure you have a one-way failover script fire when the CARP interface fails over is easy. Snapshotting the ZFS filesystems is easy. Simulating breaks by pulling a power plug is hard. Write some garbage to the FS. Unplug the DAS. Unplug the host. Force "up but non-responsive" situations, figure out how to identify them, and how to recover from them.
Doing anything reliable with iSCSI is hard. Even Amazon has a poor reputation for it. Do your best to avoid it, and never ever ever buy into vendor promises that it'll allow you to just remount your block devices on a different host and never have to worry about data or downtime. IME. YMMV.
Using runit to keep your Ruby processes up: Easy. Proper logging, monitoring and alerts because processes are just bouncing constantly? Well, it's not rocket science. But it definitely takes discipline.
There are people a hell of a lot smarter than me that have been doing this for a lot longer. But some of this stuff is just flat out not acceptable. I've been on those angry-client phone calls. "Inadvertent" should be a trigger word to anyone in Ops. It really means somebody didn't do something they should have and it bit them in the ass (at least that's what it meant when I said it). Would the app have started fine if boot.rb didn't attempt to connect to Redis? That seems pretty far fetched. So that sounds like a red-herring. But what do I know.
Setting these things up is not the hard part. Identifying that a host isn't responding to ping isn't the hard part. Planning, procedures, discipline and execution, day in and day out when things aren't on fire to prepare for the day that they are, that's the hard part.
Maybe Github is has a much smaller Ops team and budget than I would've imagined. Maybe this should be a wake up call to the CEO. I don't know. These are just my observations from what was said and having deja-vu thinking about lessons I had to learn the hard way.
BTW, I'm not necessarily trying to bag on the guy getting paged at 3AM. Been there, done that. It sucks. You do the best you can. If anything that I'm saying has a ring of truth to it, then it's a leadership issue. And it's not about "give smart people things and get out of their way blah blah bullshit". This sort of stuff doesn't materialize out of thin air and good intentions.
OTOH it's crazy that Github is single-homed and can't afford an F5. :shrug:
BTW:
> I can bet it would've taken us much longer
One of the worst experiences of my professional life was 72 sleepless hours, mostly in a 50F data-center, trying to figure out why the SAN would sporadically drop off servers every other minute. Turns out somebody, not me for a change, set the MTU on the switches to 9000. So whenever a max-frame was used BOOM.
But yeah, I've been through Redis failures. It didn't take me two hours to get things going again. Load-balancer failures. Database failures (backup sure, but never ever plan to actually use it; it's plan Z at best). NFS failures as well. Though those might be pushing it with heads that take almost 10 minutes just to reboot.
From the outside 2 hours seems like a very long time to identify and recover from a Redis failure. (Knock on wood.) And it sounds like they didn't have Warm systems standing by for failover? That's bad...
Weird that they didn't say what caused the power outage and what the mitigations are for that.
I'm also confused about how the racks would lose power. Surely they had UPSes.
> Over the past week, we have devoted significant time and effort towards understanding the nature of the cascading failure which led to GitHub being unavailable for over two hours. I don't mean to be blasphemous, but from a high level, is the performance issues with Ruby (and Rails) that necessitate close binding with Redis (i.e., lots of caching) part of the issue? It sounds like the fundamental issue is not Ruby,…
It has nothing to do with Ruby, or Rails or even Redis. It's just a design flaw of the application, that you often learn the hard way.
There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.
> There's no mention of why they don't have redundant systems in more than one datacenter sometimes reading comments on hn makes me laugh out loud. there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post: "Well, the reason we don't have an entire replica of our entire installation is because it costs way too…
So, a very small risk of an hour or so of downtime sometime in the future which will not cause data loss, or tens of thousands of dollars a month for a failover cluster? I wouldn't replicate it either.