Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

81–90 of 119 posts

Re: GitHub: October 21 Incident Report

#81

Post by twunde »

ams6110 wrote: post #66
Klathmon wrote: post #63

Earlier quoted context omitted.

with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code). We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there). In a disaster recovery scenario, th…

> In a disaster recovery scenario That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.

Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data loss should be recoverable. It's also not client-facing, but is a temporary problem similar to wifi going down at your office. An inconvenience and hassle, yes. Long-term problem, no.

Furthermore, the problem with these disaster scenarios are that there are much more dangerous problems than your account being deleted. Someone with admin access, could insert a back door or sell your source code to someone else. That's honestly scarier.

Re: GitHub: October 21 Incident Report

#82

Post by platinium »

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Which brings into question... what is so special about the "front-end" features that Git provides? Why not use another third-party service that integrates into your Github/Gitlab/Bitbucket repo, storing the meta data of all the pull requests, code reviews, discussions, etc, inside a "meta repo" itself? I see this incident as an important note in taking steps to decouple important "meta" features from the cloud-host itself. It will be better to keep the cloud hosts as a bare bones skeleton that we can swap in/out at will (Github, Gitlab, etc), and then use another layer on top to provide all the nice fancy features.

Re: GitHub: October 21 Incident Report

#83

Post by WorldMaker »

LaserToy wrote: post #75

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.

"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database".

It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce.

Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Redis partitioning: https://redis.io/topics/partitioning

Re: GitHub: October 21 Incident Report

#84

Post by kojeovo »

nik736 wrote: post #52
laumars wrote: post #14

Earlier quoted context omitted.

> Is there anything special about it? Yup. You notice when they're down. Whereas when your self-hosted git server goes offline for a couple of hours, nobody else notices.

I am not talking about git hosting per se, but compared to other SaaS companies it seems GitHub/GitLab and Bitbucket are down much more often.

[citation needed]

Re: GitHub: October 21 Incident Report

#85

Post by shaklee3 »

robinhood wrote: post #74
platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Every company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.

I think it's more about people not looking for alternatives while things are going smooth. Now they will, even though the grass my not be greener.

Re: GitHub: October 21 Incident Report

#86

Post by zzzcpan »

WorldMaker wrote: post #83
LaserToy wrote: post #75

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.

"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database". It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce. Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Re…

"Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe this problem is commonly known as split brain [1].

[1] https://en.wikipedia.org/wiki/Split-brain_(computing)

Re: GitHub: October 21 Incident Report

#87

Post by fishnchips »

platinium wrote: post #73
fishnchips wrote: post #67

Earlier quoted context omitted.

In order to objectively assess their level of professionalism, you'd need to know what happened, and what's going on in there right now. Think about it this way - can someone break things where you work right now to cause an outage of this magnitude? For all places that I've worked the answer would be "yes, of course".

It doesn't matter, really. Its a black box from a business perspective. Some users have lost faith, and some people will migrate to other solutions. Regardless of how fair or unfair the incident was, it is a fact that it was poor up-time especially for such an important cloud provider for code.

Making decisions based on a single event is risky. One could even argue that an event of this magnitude is likely to cause significant improvements in reliability. And you just paid the price for that as a user, so unless they keep failing repeatedly, you may be better off sticking with them.

Re: GitHub: October 21 Incident Report

#88

Post by sagichmal »

nik736 wrote: post #52
laumars wrote: post #14

Earlier quoted context omitted.

> Is there anything special about it? Yup. You notice when they're down. Whereas when your self-hosted git server goes offline for a couple of hours, nobody else notices.

I am not talking about git hosting per se, but compared to other SaaS companies it seems GitHub/GitLab and Bitbucket are down much more often.

I don't know about GitLab and Bitbucket, but GitHub's uptime is relatively phenomenal, given its load and feature set. I've never been in a programming environment where it's been the weakest link among hosted services.

Re: GitHub: October 21 Incident Report

#89

Post by zzzcpan »

fishnchips wrote: post #87
platinium wrote: post #73

Earlier quoted context omitted.

It doesn't matter, really. Its a black box from a business perspective. Some users have lost faith, and some people will migrate to other solutions. Regardless of how fair or unfair the incident was, it is a fact that it was poor up-time especially for such an important cloud provider for code.

Making decisions based on a single event is risky. One could even argue that an event of this magnitude is likely to cause significant improvements in reliability. And you just paid the price for that as a user, so unless they keep failing repeatedly, you may be better off sticking with them.

> One could even argue that an event of this magnitude is likely to cause significant improvements in reliability.

Doesn't happen in practice and usually the whole thing is just blamed on "process" with subsequent "process changes".

Re: GitHub: October 21 Incident Report

#90

Post by CamTin »

twunde wrote: post #81
ams6110 wrote: post #66

Earlier quoted context omitted.

> In a disaster recovery scenario That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.

Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data…

That's probably the case for valley-style startups where the whole team can fit in a room and they all hack on the same handful of repos, but most "enterprise" customers will have hundreds of repos with not necessarily anybody hacking on most of them at any given moment. It's very good policy for such organizations to have a plan in place to "break glass in case Github is down" with local mirroring of all data and a tested process for doing deploys without Github.
Post reply on HN