151–160 of 198 posts
"We identified the hardware issue resulting in servers being unable to view their own drives after power-cycling as a known firmware issue that we are updating across our fleet." Tell us which vendor shipped that firmware, so everyone else can stop buying from them.
Earlier quoted context omitted.
You could google "HA", click in the Wikipedia link that shows all the things "HA" may refer to, and deduct that the most logical thing in the list, given the context, would be this link: https://en.wikipedia.org/wiki/High_availability .
Would it have been so hard to just type "high availability" rather than making him feel bad for being one of today's 10,000? https://xkcd.com/1053/
Earlier quoted context omitted.
The most recent issue that affected me was when all EC2 instances in VPCs couldn't connect to S3. At all. It wasn't indicated on the status page until after it was fixed. And it was indicated as a green check in a sea of green checks. With a small "i" in the corner to represent the outage. I love AWS. It's not without fault but overall I think it's been well architected, well documented, and well implemented. But the…
Huh, I wonder if the status page is in fact based on any automated monitoring at all, or just manual updates? I guess probably automated monitoring, just not very good automated monitoring.
Earlier quoted context omitted.
Is there a timeline to how long it took them to figure out Redis was down? Because having experienced the same, you get an alert. Cool. HA-Proxy says app servers are down. Ok. You SSH in and see that everything looks ok but the processes are bouncing. You tail the logs to find out why (obviously lots of these steps could be optimized). Within a few seconds you spot the error connecting to Redis. A minute later you've…
Is it really "shameful"? Running systems like this at scale is hard. We're not talking about redundant power systems for an ICU Ward in a hospital. We're talking about a website which powers a sliver of the first world. You bet they busted their ass to get this fixed and shared their learnings with us. I'm extremely grateful for this and yeah it inconvenienced my morning but nothing more. You make it sound so easy. I…
Earlier quoted context omitted.
> There's no mention of why they don't have redundant systems in more than one datacenter sometimes reading comments on hn makes me laugh out loud. there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post: "Well, the reason we don't have an entire replica of our entire installation is because it costs way too…
This is not only obviously true, I think it is also a completely reasonable calculus. They just proved that if the entire Redis cluster goes down they can get it back in 2.5 hours. It's almost certainly a caching layer, so there is no permanent data loss. If they fix the application bootstrap dependency on a Redis connection, and they add monitoring to more easily see in the future when the Redis cluster is the probl…
A lot of people have started depending on github for more than just stashing source code some place centrally accessible as they're working on it. If github takes a lax attitude toward uptime then I suspect people will start looking for alternatives.
I seriously doubt this version of the story. While it's possible for several hardware/firmware to fail in all your datacenters, for them to fail at the same time is highly unlikely. This may just be a PR spin to think they're not vulnerable to security attacks. While this was happening at Github, I noticed several other companies facing that same issue at the same time. Atlassian was down for the most part. It could…
They're not hosted in multiple datacenters; there was a power interruption in their single datacenter that exposed this firmware bug. The point of this postmortem isn't the initial power interruption but rather its repercussions, why it took so long to recover from and how they can improve their response and communications in the future.
Earlier quoted context omitted.
Would it have been so hard to just type "high availability" rather than making him feel bad for being one of today's 10,000? https://xkcd.com/1053/
Why would you feel bad about not being familiar with an abbreviation?
Earlier quoted context omitted.
This is not only obviously true, I think it is also a completely reasonable calculus. They just proved that if the entire Redis cluster goes down they can get it back in 2.5 hours. It's almost certainly a caching layer, so there is no permanent data loss. If they fix the application bootstrap dependency on a Redis connection, and they add monitoring to more easily see in the future when the Redis cluster is the probl…
Maybe they do feel it's a reasonable business decision. In that case they shouldn't be surprised if a lot of their users make the equally reasonable business decision to reduce their exposure to Github. A lot of people have started depending on github for more than just stashing source code some place centrally accessible as they're working on it. If github takes a lax attitude toward uptime then I suspect people wil…
For all that work to be done in just two hours is amazing, especially with degraded internal tools, and both hardware and ops teams working simultaneously.
So thank you GitHub, please keep up the good work!