Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

21–30 of 198 posts

Re: January 28th Incident Report

#21

Post by merqurio »

davidcelis wrote: post #11
merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

Was it a good incident to see how dependent we are on GitHub? Every time there's a GitHub outage, a vocal group of people will voice their opinions that we are too dependent on GitHub, we should be using open source alternatives, GitHub should be open source, etc. Then, within a few days, everybody goes silent and we return to our normal lives. I don't think outages at GitHub are very frequent. This one was lengthy,…

Of course it was. I don't know if everybody goes silent after a few days, It's the first outage I'm aware of, but some people at university made see the hypocrisy of using GitHub for open source projects and I feel that if there is a community strong enough to make some impact on GitHub that could be hackernews. Maybe I'm wrong.

Re: January 28th Incident Report

#22

Post by DarkTree »

I don't know enough about server infrastructure to comment on whether or not Github was adequately prepared or reacted appropriately to fix the problem.

But wow it is refreshing to hear a company take full responsibility and own up to a mistake/failure and apologize for it.

Like people, all companies will make mistakes and have momentary problems. It's normal. So own up to it and learn how to avoid the mistake in the future.

Re: January 28th Incident Report

#23

Post by davidcelis »

merqurio wrote: post #21
davidcelis wrote: post #11

Earlier quoted context omitted.

Was it a good incident to see how dependent we are on GitHub? Every time there's a GitHub outage, a vocal group of people will voice their opinions that we are too dependent on GitHub, we should be using open source alternatives, GitHub should be open source, etc. Then, within a few days, everybody goes silent and we return to our normal lives. I don't think outages at GitHub are very frequent. This one was lengthy,…

Of course it was. I don't know if everybody goes silent after a few days, It's the first outage I'm aware of, but some people at university made see the hypocrisy of using GitHub for open source projects and I feel that if there is a community strong enough to make some impact on GitHub that could be hackernews. Maybe I'm wrong.

If you look back through the years and find a few other stories of "GitHub is down", you'll see that this conversation happens every time. Some people tread into the HackerNews thread and say "More people should be using self-hosted GitLab instances" or "if GitHub would just open source their code, we wouldn't need to be so dependent."

But then the conversation stops within days because, the fact is, hosting your own git servers and getting people to actually use them is a huge pain in the ass. More simply put: people just like using GitHub.

Furthermore, GitHub's a business. They're selling private repositories. They do open source quite a bit of code, but they're not going to open source their actual product.

Re: January 28th Incident Report

#24

Post by yeukhon »

viraptor wrote: post #14
dsmithatx wrote: post #6

If only Bitbucket could give such comprehensive reports. A few months back outages seemed almost daily. Things are more stable now. I hope for the long term.

Isn't BB's problem basically that there are too many users? GH's outage writeup is cool, because it's a one off and it can be analysed. When BB is just overloaded for a long time and needs more power, it's not going to be very interesting. (unless I missed some specific non capacity related outages?)

Maybe. BitBucket was also an acquisition so for some time I believe there was a lack of resource provided to them and there was a huge technical debt/integration effort required. At this very time, I don't know if Atlassian actually care much about BitBucket. They are probably more concerned about delivering Stash than BitBucket, my wild guess.

I was an active BB user a couple years ago, and the project I worked on would hg clone from BB many times a day so I would be the first one to notice a 503 or whatever error coming from their service. Typically I would see one or two outage per month, some last a few minutes, some last several hours. Most of the time the outage impacted git/hg checkout, so I think that was their technical bottleneck.

Re: January 28th Incident Report

#25

Post by alblue »

swrobel wrote: post #17

Anyone got a good tl;dr version?

Power outage in DC brought many machines down. Redis clusters failed to start owing to disk issues (not cleanly unmounted?). The reboot of remaining machines uncovered an unknown dependency on the machines needing the redis cluster to be up in order to boot.

There were other learning points such as immediately going into anti DDoS mode and human communication issues that didn't realise or escalate the problem until some time after the issues started occurring.

Re: January 28th Incident Report

#26

Post by jdboyd »

merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

If github opensource all of their stuff, this still wouldn't prevent issues like this for the projects that want to use a hosted service instead of hosting it themselves, and many projects don't want to host these services them selves anymore.

When I worry about dependency on GitHub, I'm thinking about not the inconvenient hours of downtime but the larger threat that they might disappear or turn evil.

What I would like to see even more than opensource github would be a standard for spreading over more services. For instance, syncing code, issues, pull requests, wiki, pages, etc between self-hosted gitlab and gitlab.com, or between gitlab.com and github.com. Further, I'd like to see it be easier to use common logins across services.

I don't think we can rely on Github giving us this, but if GitLab would add it between gitlab.com and gitlab ce, that would be a compelling reason to think of switching.

Re: January 28th Incident Report

#28

Post by BinaryIdiot »

merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy.

So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If you want open source use Gitlab or any other service that sprinkles extra features around git.

I'll never understand this outrage of dependence on Github when you have a distributed version control system. It's not like it should be on github to setup third party repositories for you.

Re: January 28th Incident Report

#29

Post by jtblin »

nemothekid wrote: post #7
bosdev wrote: post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

> Given the dependency in question is Redis, such a solution is probably exasperated by the fact Redis hasn't really had a decent HA solution.

Redis sentinel[0] is the HA solution for redis for quite some time.

[0]http://redis.io/topics/sentinel

Post reply on HN