Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

121–130 of 198 posts

Re: January 28th Incident Report

#121

Post by nissehulth »

BinaryIdiot wrote: post #28
merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy. So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If yo…

From a developers point of view, you're right. But there are package management systems and other stuff depending on being able to download from Github.

Ofc, Github isn't to blame for this, rather the ones that thought Github would be great to use as a CDN.

Re: January 28th Incident Report

#122

Post by fapestniegd »

tonylxc wrote: post #71
theptip wrote: post #69

Earlier quoted context omitted.

The rest of the sentence is pertinent: "...but we can take steps to ensure recovery occurs in a fast and reliable manner. We can also take steps to mitigate the negative impact of these events on our users." The lessons that giants like Netflix have learned about running massive distributed applications show that you cannot avoid failure, and instead must plan for it. Now, having a single datacenter is not a good pla…

My point is: they shouldn't ONLY plan on ensuring recovery occurs fast; they should also plan on having multiple data centers, which to me is more important. It's frightening to know that such an important service is only operating in a single data center. However, their recovery report didn't mention anything about such a plan. << Edited: correct a grammar error.

You're not going to fail over to a secondary datacenter in under 125 minutes. A RTO under that costs a prohibitively stupid amount.

Re: January 28th Incident Report

#123

Post by spydum »

Someone1234 wrote: post #63
spydum wrote: post #61

So, while it sounds like they have reasonable HA, they fell down on DR. unrelated, I could not comprehend what this means?..: technicians to bring these servers back online by draining the flea power to bring Flea power?

I assume they mean completely disconnect the equipment from ALL external power sources. Typically even when a piece of equipment is offline in a data center, it continues to draw power, and will often keep running systems like DRAC and other management/status tools (since the whole concept of a data center is NEVER having to get up out of your chair, so even a "shutdown" system needs to be able to be remotely started…

sure enough, it's a real term, and it's relatively old.. http://answers.google.com/answers/threadview/id/185999.html

I have never known what to call this, but have definitely been engaged in draining a few fleas.

Also, I can't believe it's been that long since google answers has been closed..

Re: January 28th Incident Report

#125

Post by reustle »

abrookewood wrote: post #116
bmurphy1976 wrote: post #114

Earlier quoted context omitted.

I take it you have no experience filing tickets with them. A typical ticket goes something like this: 1. File ticket. 2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour). 3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalli…

To be honest, I've always found their support to be really good. Sometimes it can be a little slow to start, but I regularly experience technicians that go way above what I would expect to assist me & deliver a great outcome. If other companies in Australia were as responsive as them (e.g. telcos), I'd be a very happy man. EDIT: I'm on Business Support, so maybe that's your issue?

> Sometimes it can be a little slow to start

Which is unacceptable for one of the largest infrastructure providers. So many times we were sitting around twiddling our thumbs waiting for our expensive amazon support to get back to us when things were broken.

Re: January 28th Incident Report

#126

Post by BinaryIdiot »

nissehulth wrote: post #121
BinaryIdiot wrote: post #28

Earlier quoted context omitted.

Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy. So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If yo…

From a developers point of view, you're right. But there are package management systems and other stuff depending on being able to download from Github. Ofc, Github isn't to blame for this, rather the ones that thought Github would be great to use as a CDN.

I agree with you that is an issue for the package manager. A package manager should never have such a single point of failure. I know bower, for example, would fail (at least last I checked unless they added a caching system since then) but it at least can point to specific git repositories so it can be moved if necessary.

Honestly if anyone is working on something important enough where they must be able to rebuild at a moment's notice then they should either be checking in dependencies from these package managers or setting up their own copies of what they need. But that's like backing up; most don't realize they needed to be doing that until they have an issue so I can understand the anger when Github does go down.

Re: January 28th Incident Report

#127

Post by jsjohnst »

abrookewood wrote: post #116
bmurphy1976 wrote: post #114

Earlier quoted context omitted.

I take it you have no experience filing tickets with them. A typical ticket goes something like this: 1. File ticket. 2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour). 3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalli…

To be honest, I've always found their support to be really good. Sometimes it can be a little slow to start, but I regularly experience technicians that go way above what I would expect to assist me & deliver a great outcome. If other companies in Australia were as responsive as them (e.g. telcos), I'd be a very happy man. EDIT: I'm on Business Support, so maybe that's your issue?

I'm on business support too and generally am talking to a rep in minutes. They aren't always able to find the problem before I do, but I always get follow up details later on the how / why that they did determine.

Re: January 28th Incident Report

#129

Post by paulddraper »

click170 wrote: post #91
atom_enger wrote: post #89

Earlier quoted context omitted.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…

If you have a support agreement with them then file a ticket requesting better customer communication and link back here as an example of how to do it right. I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.

I've filed tickets about their status page before, especially on the stupid green-checkmark-with-i.

Hasn't worked yet.

Re: January 28th Incident Report

#130

Post by paulddraper »

atom_enger wrote: post #89
dmunoz wrote: post #86

Earlier quoted context omitted.

I recently stepped into a role with a devops component, and one of my first surprises was just how slow status.aws.amazon.com was to update about ongoing issues. I had to scramble to find twitter and external forums confirmation for the client.

What's even worse is that when Amazon finally updates their status page it's usually still a green icon with a little i tick for "information" even if it was a partial outage. It takes a lot for the icons to go red which is what you'd look for if you're experiencing issues. I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It…

The most recent issue that affected me was when all EC2 instances in VPCs couldn't connect to S3. At all.

It wasn't indicated on the status page until after it was fixed. And it was indicated as a green check in a sea of green checks. With a small "i" in the corner to represent the outage.

I love AWS. It's not without fault but overall I think it's been well architected, well documented, and well implemented.

But the status page has got to be the ultimate example of what not to do.

Post reply on HN