Live data from Hacker News

January 28th Incident Report

github.com

131–140 of 198 posts

Re: January 28th Incident Report

#131
post #98
post #67

Earlier quoted context omitted.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

For people reading this, AZ in this context would be Availability Zone

Right, and not the Grand Canyon State.

The space of acronyms/abbreviations is quite cluttered.

Re: January 28th Incident Report

#132
post #51

TL;DR: "We don’t believe it is possible to fully prevent the events that resulted in a large part of our infrastructure losing power, ..." This doesn't sound very good.

Whose datacenter are they in? This is the second time in less than two weeks that they've suffered a power-related issue. My company is in 4 different sites around the world and we've never lost power ever - and, if one circuit did go out, we'd still be up and running because all of our servers have redundant power supplies on separate infeed circuits.

Re: January 28th Incident Report

#133
post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

Seriously. I'm kind of surprised about this.

Yeah, they gloss over it but at its heart, keeping mission-critical servers in a single datacenter with no redundancy is among the most common and amateur infrastructure failures. Many would expect a company like GitHub to have anticipated and prevented it. GitHub should have a process to ensure that all services are redundant before they get pushed to production.

Re: January 28th Incident Report

#134
post #98
post #67

Earlier quoted context omitted.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

For people reading this, AZ in this context would be Availability Zone

[deleted]

Re: January 28th Incident Report

#135
post #98
post #67

Earlier quoted context omitted.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

For people reading this, AZ in this context would be Availability Zone

Right, and not the Grand Canyon State.

The space of acronyms/abbreviations is quite cluttered.

Re: January 28th Incident Report

#136

Earlier quoted context omitted.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…

The most recent issue that affected me was when all EC2 instances in VPCs couldn't connect to S3. At all. It wasn't indicated on the status page until after it was fixed. And it was indicated as a green check in a sea of green checks. With a small "i" in the corner to represent the outage. I love AWS. It's not without fault but overall I think it's been well architected, well documented, and well implemented. But the…

Huh, I wonder if the status page is in fact based on any automated monitoring at all, or just manual updates? I guess probably automated monitoring, just not very good automated monitoring.

Re: January 28th Incident Report

#137

Earlier quoted context omitted.

> There's no mention of why they don't have redundant systems in more than one datacenter sometimes reading comments on hn makes me laugh out loud. there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post: "Well, the reason we don't have an entire replica of our entire installation is because it costs way too…

This is not only obviously true, I think it is also a completely reasonable calculus. They just proved that if the entire Redis cluster goes down they can get it back in 2.5 hours. It's almost certainly a caching layer, so there is no permanent data loss. If they fix the application bootstrap dependency on a Redis connection, and they add monitoring to more easily see in the future when the Redis cluster is the probl…

>It's almost certainly a caching layer, so there is no permanent data loss.

People who use Redis rarely end up using it solely as a caching layer. It often also takes on the role of an RPC facilitator and pseudo-database. GitHub's post also mentions that their engineering team had to replicate Redis' dataset before they could get the alternative hardware running, which implies that they do need some data in there before the site is operational.

Personally one of my pet peeves is people throwing mission-critical data in Redis and acting like it's honky-dory. It happens all the time and seems really difficult to get people to not do. There's a reason we have a real ACID compliant database storing non-disposable data; it's ridiculous to ignore that just because it's easier to stuff it in Redis.

I think it's reasonable to have a dependency on a Redis server, but I don't think it's reasonable to depend on any data in particular being stored in that server. It should be used as a caching/acceleration layer for data that can be easily and automatically regenerated.

Re: January 28th Incident Report

#138

Earlier quoted context omitted.

> There's no mention of why they don't have redundant systems in more than one datacenter sometimes reading comments on hn makes me laugh out loud. there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post: "Well, the reason we don't have an entire replica of our entire installation is because it costs way too…

This is not only obviously true, I think it is also a completely reasonable calculus. They just proved that if the entire Redis cluster goes down they can get it back in 2.5 hours. It's almost certainly a caching layer, so there is no permanent data loss. If they fix the application bootstrap dependency on a Redis connection, and they add monitoring to more easily see in the future when the Redis cluster is the probl…

[deleted]

Re: January 28th Incident Report

#139

Earlier quoted context omitted.

To be honest, I've always found their support to be really good. Sometimes it can be a little slow to start, but I regularly experience technicians that go way above what I would expect to assist me & deliver a great outcome. If other companies in Australia were as responsive as them (e.g. telcos), I'd be a very happy man. EDIT: I'm on Business Support, so maybe that's your issue?

I'm on business support too and generally am talking to a rep in minutes. They aren't always able to find the problem before I do, but I always get follow up details later on the how / why that they did determine.

I wish our experience was like this. We used to have business level but we dropped it because we weren't getting value for it. Our experience was slightly better when we had it but we still ended up either fixing most problems on our own or waiting them out.

Re: January 28th Incident Report

#140

Earlier quoted context omitted.

No system is perfect; as you continue to add 9s, the cost increases steeply. Usually its just cheaper to be down for an hour or two, versus architect for the end of times.

> Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.

[deleted]
Post reply on HN