Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

101–110 of 119 posts

Re: GitHub: October 21 Incident Report

#101

Post by gumby »

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

> Actually, I don't know why I pay for Github private repo's.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal.

You do realize that every cloned tree can be a git "repo", regardless of the machine it's on, right? GH and GL surround the repo with some other things (bug reports etc) to rope you in but if you are cloning from one to the other you already aren't migrating that stuff as well, so it's not really clear what additional value that provides.

Is there something I'm not seeing?

Re: GitHub: October 21 Incident Report

#102

Post by module0000 »

CamTin wrote: post #90
twunde wrote: post #81

Earlier quoted context omitted.

Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data…

That's probably the case for valley-style startups where the whole team can fit in a room and they all hack on the same handful of repos, but most "enterprise" customers will have hundreds of repos with not necessarily anybody hacking on most of them at any given moment. It's very good policy for such organizations to have a plan in place to "break glass in case Github is down" with local mirroring of all data and a…

We have exactly that... in our DR plan, there is a section for how to cope with the 3rd party source control provider being unavailable/compromised/etc. Update DNS for the equivalent of "upstream-git.foo.com" to an internal address, and continue business as usual.

It's like you said, smaller shops probably think it's over-planning and overkill, but we do indeed have 100's of projects that are "mission critical", that might not have been touched in 1+ years.

Re: GitHub: October 21 Incident Report

#103

Post by jhabdas »

gumby wrote: post #101
platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

> Actually, I don't know why I pay for Github private repo's.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal. You do realize that every cloned tree can be a git "repo", regardless of the machine it's on, right? GH and GL surround the repo with some other things (bug reports etc) to rope you in but if you are cloning from o…

Save yourself some money and store your personal private repos encrypted on Keybase or a self-hosted solution.

Re: GitHub: October 21 Incident Report

#104

Post by limonkufu »

elliotlarson wrote: post #57
colonelpopcorn wrote: post #56

Earlier quoted context omitted.

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

I am currently leading the on premise hosting of a Gitlab instance and I can say this with ease: I have been spending 1 day of each week for ops work. Let it be helping people, database adjustments, admin stuff, hardware checks, etc.

Re: GitHub: October 21 Incident Report

#106

Post by sneak »

GitHub also silently published a private repo of mine this week. I checked audit logs for both the owner user and the org and it didn’t show a permissions change anywhere.

Netlify has caused me to stop using GitHub Pages and between the clownshoes outage reports and the security issue I am now a GitLab user.

This is GitHub’s jump the shark episode. :(

Re: GitHub: October 21 Incident Report

#107

Post by andyana »

sneak wrote: post #106

GitHub also silently published a private repo of mine this week. I checked audit logs for both the owner user and the org and it didn’t show a permissions change anywhere. Netlify has caused me to stop using GitHub Pages and between the clownshoes outage reports and the security issue I am now a GitLab user. This is GitHub’s jump the shark episode. :(

Gitlab has had its own issues.

Re: GitHub: October 21 Incident Report

#108

Post by simlevesque »

robinhood wrote: post #74
platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Every company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.

Why does GitHub need my support ?

Re: GitHub: October 21 Incident Report

#109

Post by codefined »

PuffinBlue wrote: post #100
ikornaselur wrote: post #98

Earlier quoted context omitted.

> GitLab infamously lost their entire production database What?? The link you posted says > Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately. > It's hard to estimate how much data has been lost exactly, but we estimate we have lost at least 5000 projects, 5000 comments, a…

Technically they did, for a brief period (24 hours??) at least, they just restored it from backup. That was the final last ditch backup too where something like 5 out of 6 of the planned backups weren't actually working and nobody realised. So you're right, they didn't lose it, but they came pretty damn close!

Picture of the Google Docs post on what they tried (which appears to no longer be available online) can be found:

https://femto.pw/6bsm.png

As a reference to the above poster.

Re: GitHub: October 21 Incident Report

#110

Post by mr_toad »

Something that struck me yesterday as this started, is that Github isn’t really just a dvcs hosting solution, Github is a social network.

It’s easy to change the remote origin of a git repository. It’s not hard to migrate a project to Gitlab etc, and duplicate all the technical features of Github.

What’s really hard is replacing the social graph. If you’re a large project with a lot of contributors, onboarding everyone is not going to be easy. About as easy as convincing all your friends to stop using Facebook.

Post reply on HN