Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

91–100 of 198 posts

Re: January 28th Incident Report

#91

Post by click170 »

atom_enger wrote: post #89
dmunoz wrote: post #86

Earlier quoted context omitted.

I recently stepped into a role with a devops component, and one of my first surprises was just how slow status.aws.amazon.com was to update about ongoing issues. I had to scramble to find twitter and external forums confirmation for the client.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…

If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right.

I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.

Re: January 28th Incident Report

#92

Post by atom_enger »

ssmoot wrote: post #60
danielvf wrote: post #9

For all that work to be done in just two hours is amazing, especially with degraded internal tools, and both hardware and ops teams working simultaneously.

Is there a timeline to how long it took them to figure out Redis was down? Because having experienced the same, you get an alert. Cool. HA-Proxy says app servers are down. Ok. You SSH in and see that everything looks ok but the processes are bouncing. You tail the logs to find out why (obviously lots of these steps could be optimized). Within a few seconds you spot the error connecting to Redis. A minute later you've…

Is it really "shameful"? Running systems like this at scale is hard. We're not talking about redundant power systems for an ICU Ward in a hospital. We're talking about a website which powers a sliver of the first world.

You bet they busted their ass to get this fixed and shared their learnings with us. I'm extremely grateful for this and yeah it inconvenienced my morning but nothing more.

You make it sound so easy. If it takes the Github folks 2 hours, I can bet it would've taken us much longer.

Re: January 28th Incident Report

#93

Post by edem »

BinaryIdiot wrote: post #28
merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy. So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If yo…

During the outage I was not able to push and that's all. I can build everything locally and push to Heroku from my machine. You can always add an upstream repository to Bitbucket or something like that.

Re: January 28th Incident Report

#94

Post by eric_h »

dmunoz wrote: post #86
eric_h wrote: post #83

> One of the biggest customer-facing effects of this delay was that status.github.com wasn't set to status red until 00:32am UTC, eight minutes after the site became inaccessible. We consider this to be an unacceptably long delay, and will ensure faster communication to our users in the future. Amazon could learn a thing or two from Github in terms of understanding customer expectations.

I recently stepped into a role with a devops component, and one of my first surprises was just how slow status.aws.amazon.com was to update about ongoing issues. I had to scramble to find twitter and external forums confirmation for the client.

day to day i mostly write software, but I also help manage our infrastructure (we're a small company - 9 people total, 4 engineers, I'm one of the 2 that understands managing servers well enough to support it). We were on linode up until about a year and change ago and switched to AWS/Opsworks to both decrease our infrastructure bill and increase our ability to scale horizontally quickly (for unfortunately long definitions of quickly - "running setup...")

Both Linode and Amazon suck at their status pages (though linode was quite informative about their DDoS outages that started on Christmas). Every amazon issue we've had, the status page only changed once they'd more or less fixed it. As far as I'm concerned their status page is basically useless unless it's an extended outage, at which point it's still basically useless...

Re: January 28th Incident Report

#96

Post by eric_h »

click170 wrote: post #91
atom_enger wrote: post #89

Earlier quoted context omitted.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…

If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.

I think a lot of folks feel that it's a useless endeavor, so they don't bother. Amazon's been operating this way for years, and they're quite a large company; it seems unlikely to me that fundamental change can happen inspired by customer tickets, even if you're paying for support.

Basically, if Netflix isn't the source of the complaint, they're not going to give two fucks.

/me suspects that netflix engineers get outage notifications through some other avenue than the status page.

Re: January 28th Incident Report

#97

Post by atom_enger »

tmsh wrote: post #68

> Over the past week, we have devoted significant time and effort towards understanding the nature of the cascading failure which led to GitHub being unavailable for over two hours. I don't mean to be blasphemous, but from a high level, is the performance issues with Ruby (and Rails) that necessitate close binding with Redis (i.e., lots of caching) part of the issue? It sounds like the fundamental issue is not Ruby,…

I don't think that Ruby/Rails has anything to do with this, really. If you want to scale any app, you're going to want to do some caching somewhere. What this boils down to is that their app has a dependency in an initializer that depends on redis. Without a connection to redis, it will flap.

Re: January 28th Incident Report

#98

Post by drdrey »

theptip wrote: post #67
bosdev wrote: post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

For people reading this, AZ in this context would be Availability Zone

Re: January 28th Incident Report

#99

Post by obituary_latte »

dice wrote: post #90
obituary_latte wrote: post #79

Earlier quoted context omitted.

Not sure how your comment refutes the contention of reliability. Seems to me to be more a condemnation of failures that do happen (which is of course worthy of concern, but irrelevant in a conversation about stability).

I am using "reliability" in the sense of RAS[0]. An HA datastore which erroneously ACK's writes has lowered reliability, as there are known cases where it gives incorrect outputs. 0: https://en.wikipedia.org/wiki/Reliability,_availability_and_...

I see. Yeah, more concerning there being errors not based on an "event". Thanks for clarifying; sorry for any confusion.
Post reply on HN