Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: GitHub: October 21 Incident Report

#61

Post by module0000 »

elliotlarson wrote: post #57
colonelpopcorn wrote: post #56

Earlier quoted context omitted.

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.

Re: GitHub: October 21 Incident Report

#62

Post by [deleted] »

colonelpopcorn wrote: post #56
Klathmon wrote: post #53

Earlier quoted context omitted.

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's…

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

[deleted]

Re: GitHub: October 21 Incident Report

#63

Post by Klathmon »

module0000 wrote: post #61
elliotlarson wrote: post #57

Earlier quoted context omitted.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.

with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code).

We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there).

In a disaster recovery scenario, that's more than enough for me.

Sure, if github blinked out of existence we would probably be at a fraction of our normal productivity for a while until we fully recover everything and find new workflows, but the risk vs reward there is well within the margins of what I'd consider acceptable for a company like us.

Re: GitHub: October 21 Incident Report

#65

Post by mk89 »

module0000 wrote: post #61
elliotlarson wrote: post #57

Earlier quoted context omitted.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.

>but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror.

For me it's actually the opposite, except for one company that had only an internal SCM and no cloud stuff.

Re: GitHub: October 21 Incident Report

#66

Post by ams6110 »

Klathmon wrote: post #63
module0000 wrote: post #61

Earlier quoted context omitted.

Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.

with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code). We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there). In a disaster recovery scenario, th…

> In a disaster recovery scenario

That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.

Re: GitHub: October 21 Incident Report

#67

Post by fishnchips »

smilesmile2018 wrote: post #64

GitHub team seems to be VERY unprofessional. 15 hour outage means ~99.82% availability which is extremely bad. 9 hours ago they also told that they would fix the problem within 2 hours... still not fixed!!!

In order to objectively assess their level of professionalism, you'd need to know what happened, and what's going on in there right now. Think about it this way - can someone break things where you work right now to cause an outage of this magnitude? For all places that I've worked the answer would be "yes, of course".

Re: GitHub: October 21 Incident Report

#68

Post by platinium »

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's:

https://stackoverflow.com/questions/11690709/can-a-project-h...

  Quoted:
  "Try adding a remote called "github" instead:

   $ git remote add github    
   https://github.com/Company_Name/repository_name.git

   # push master to github
   $ git push github master

   # Push my-branch to github and set it to track github/my-branch
   $ git push -u github my-branch

   # Make some existing branch track github instead of origin
   $ git branch --set-upstream other-branch github/other-branch"
Actually, I don't know why I pay for Github private repo's.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal.

Re: GitHub: October 21 Incident Report

#69

Post by goykasi »

alexeiz wrote: post #45
edejong wrote: post #33

Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project d…

But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time.

Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand still as far as merging to master and closing out branches. Github being down had a huge affect on the process -- review, merging and CI success. Merging was our big one since we have protected branches via github -- master; CI was second since we received no webhook events. So either we wait it out (~8 hours) or we throw out our process and do something different until they fix it. The later didnt seem like a reasonable course of action.

github != git

Its easy to say, "well they are down, that doesnt affect git", but the reality is that a lot of orgs dont just use git. They use github. That fact envelopes a lot of process, routine, infra, schedule, money, etc. Luckily, its only been one day. But developer time has definitely beem lost. You cant reasonably say otherwise if you use github.

edit: I will say I dont fully agree with the GP. Blaming the downtime on MySQL is silly and coming up with dollars lost is over the top. Things happen. That comment was a weird attack on a reasonably stable database. Github had a bad day; it happens.

Post reply on HN