Live data from Hacker News

GitHub was down

status.github.com

71–80 of 197 posts

Re: GitHub was down

#71
post #70

Earlier quoted context omitted.

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

I think the point is git is becoming the new svn for many people.

Re: GitHub was down

#72
I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We purchased systems from Sun because of the quality of the HW and the stability of Solaris. Now a stack of cheap linux boxes with the right design (IP CLOS, BGP fabric, load balancing and containers) who cares if a switch fails or a rack goes out? Most issues today are caused my humans and bugs from software upgrades for new code.

Re: GitHub was down

#73
post #37

Earlier quoted context omitted.

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

[deleted]

Re: GitHub was down

#74
post #60
post #57

Everyone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc. Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.

They offer a local instance of GitHub actually.

Which I can't imagine to be any more reliable than hosted GitHub.

But hey, I guess there's the comfort in know that when it goes down it's our fault!

Re: GitHub was down

#75

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

Kind of?

How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape.

Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

Re: GitHub was down

#76
post #70

Earlier quoted context omitted.

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.

Re: GitHub was down

#77
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.

Exactly. There are no right/wrong or absolutes. It's a trade-off.. if it makes sense in someone's situation, it makes sense.

Re: GitHub was down

#78
post #59
post #37

Earlier quoted context omitted.

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…

Thing is, runtime dependencies on Github is something that Github is promoting and selling now ( https://github.com/marketplace/ ). Maybe this incident puts that value proposition in a more visceral context.

How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.

Re: GitHub was down

#79

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

Sounds like an argument against micro services?

Re: GitHub was down

#80
post #37

Earlier quoted context omitted.

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.
Post reply on HN