Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: GitHub: October 21 Incident Report

#91

Post by marcosdumay »

elliotlarson wrote: post #57
colonelpopcorn wrote: post #56

Earlier quoted context omitted.

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

You know, the time required to host Gittea (or Gogs) is basically nil (you'll have to set your environment up for Github too). And by maintaining it, you mean making sure it's online and taking backups? Because I don't see how you can save any time on those by using Github.

Where you host it is of less relevance, because you can simply take your backup and server script and run them at any different provider any time you want.

Re: GitHub: October 21 Incident Report

#92

Post by WorldMaker »

zzzcpan wrote: post #86
WorldMaker wrote: post #83

Earlier quoted context omitted.

"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database". It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce. Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Re…

"Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe…

It's not "unrelated", it's an intertwined phenomenon, especially in practical consequences. In Redis, as one primary example I already mentioned, all database partitioning is network partitioning (and vice versa). The logical object is still a "database" even if the physical object suffering problems is the "network".

For similar reasons that you suppose they "don't use proper distributed algorithms" (which seems an overly harsh way to put it), I simply presumed the logical entity in question is still a "database", even in the case of a network partition problem. I don't think it was either Postgres or Redis in this case, but they are simple enough examples to illustrate the overall problem.

Either way, my case still seems to stand that we aren't likely to get more information about the specifics of the partition failure because likely both the logical (which "database") and physical layer specifics (which datacenters/"network") are things that are internal to GitHub that they may not be able to publicly describe much more than what they have.

Re: GitHub: October 21 Incident Report

#93

Post by ProAm »

_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over,

Welcome to the SAAS world. Where you offload institutional knowledge and hiring to a 3rd party and roll the dice.

Re: GitHub: October 21 Incident Report

#95

Post by LaserToy »

WorldMaker wrote: post #92
zzzcpan wrote: post #86

Earlier quoted context omitted.

"Network partition" is unrelated to partial database failure or database partitioning. It means that database servers got disconnected from each other. Normally this shouldn't be a problem, but as far as I know they don't use proper distributed algorithms and so it's possible that disconnected servers each became masters and were serving requests independently, which is why they speak about inconsistencies. I believe…

It's not "unrelated", it's an intertwined phenomenon, especially in practical consequences. In Redis, as one primary example I already mentioned, all database partitioning is network partitioning (and vice versa). The logical object is still a "database" even if the physical object suffering problems is the "network". For similar reasons that you suppose they "don't use proper distributed algorithms" (which seems an…

It doesn’t really matter what they used, I just pointed out that their incident report is BS. I can blame virtually any production problem on network, but the ugly truth is - it is not networks fault in most of cases, but fault of engineers who developed the system with unreasonable assumptions.

Re: GitHub: October 21 Incident Report

#96

Post by Shank »

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

> this disastrous calamity by Github really made me try out Gitlab and in the process

Every company faces problems like this. GitLab infamously lost their entire production database [0] -- and I think we can agree that's more serious. Knee-jerk reactions to incidents will leave you without any "trusted" services, because mistakes happen to everybody at some point. BitBucket has certainly had its own fair share of downtime.

[0]: https://about.gitlab.com/2017/02/10/postmortem-of-database-o...

Re: GitHub: October 21 Incident Report

#97

Post by LaserToy »

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Based on what I read on their engineering blog they will eventually have a similar outage.

Re: GitHub: October 21 Incident Report

#98

Post by ikornaselur »

Shank wrote: post #96
platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

> this disastrous calamity by Github really made me try out Gitlab and in the process Every company faces problems like this. GitLab infamously lost their entire production database [0] -- and I think we can agree that's more serious. Knee-jerk reactions to incidents will leave you without any "trusted" services, because mistakes happen to everybody at some point. BitBucket has certainly had its own fair share of dow…

> GitLab infamously lost their entire production database

What?? The link you posted says

> Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately.

> It's hard to estimate how much data has been lost exactly, but we estimate we have lost at least 5000 projects, 5000 comments, and roughly 700 users. This only affected users of GitLab.com, self-hosted instances or GitHost instances were not affected.

How is that "their entire production database"? You make it sound so much worse than it was. While it was a horrible incident, they did not lose their whole production database.

Re: GitHub: October 21 Incident Report

#99

Post by ashelmire »

nik736 wrote: post #52
laumars wrote: post #14

Earlier quoted context omitted.

> Is there anything special about it? Yup. You notice when they're down. Whereas when your self-hosted git server goes offline for a couple of hours, nobody else notices.

I am not talking about git hosting per se, but compared to other SaaS companies it seems GitHub/GitLab and Bitbucket are down much more often.

Facebook had an outage recently, Google has had outages. Hulu has an outage every week. You can check https://downdetector.com/ for recent history of many major services.

It's a fair bet that developers are more likely to notice and complain about software services being down on the internet.

Re: GitHub: October 21 Incident Report

#100

Post by PuffinBlue »

ikornaselur wrote: post #98
Shank wrote: post #96

Earlier quoted context omitted.

> this disastrous calamity by Github really made me try out Gitlab and in the process Every company faces problems like this. GitLab infamously lost their entire production database [0] -- and I think we can agree that's more serious. Knee-jerk reactions to incidents will leave you without any "trusted" services, because mistakes happen to everybody at some point. BitBucket has certainly had its own fair share of dow…

> GitLab infamously lost their entire production database What?? The link you posted says > Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately. > It's hard to estimate how much data has been lost exactly, but we estimate we have lost at least 5000 projects, 5000 comments, a…

Technically they did, for a brief period (24 hours??) at least, they just restored it from backup.

That was the final last ditch backup too where something like 5 out of 6 of the planned backups weren't actually working and nobody realised.

So you're right, they didn't lose it, but they came pretty damn close!

Post reply on HN