Live data from Hacker News

GitHub was down

status.github.com

151–160 of 197 posts

Re: GitHub was down

#151
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Use Docker.

When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.

Re: GitHub was down

#152
post #133
post #118

Earlier quoted context omitted.

Uptime % includes time to recovery. Since you're not up if you're still recovering...

I think GP's point is that you should assume the 0.{however many 0s}1% it's down is going to be a bad time for you, and then ask yourself how long you want to wait to be back up. A long down time might be worse for you than a shorter down time more frequently.

I fail to see how.

99.99999999% uptime is 3 milliseconds of downtime a year. That's not even a TCP retransmission.

Re: GitHub was down

#153

Such incidents remind me of 2 things 1. No matter what 99.99999999% availability a service provides, its utterly useless if the time to get back up is unacceptable. 2. Do not have remote services, which you cannot fully control to be a part of your run time deployments.

I can't understand why people include GitHub as a deploy time dependency. Deploying should be about pushing an already built package to the destination server(s).

The idea that you push the source code then build it on the server makes no sense to me.

If I was github I would forbid (and punish) this kind of behavior.

Re: GitHub was down

#155
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Self hosted bitbucket and the like really. Or AWS CodeCommit. We are currently using both of those in our build pipelines.

Re: GitHub was down

#157
post #57

Everyone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc. Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.

> Making your organisation too dependent on a remote service can indeed be a scary prospect

At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail.

I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

Re: GitHub was down

#158
post #37
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…

> you can instead push your artifacts to an s3 target

This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).

Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.

Re: GitHub was down

#159
post #29

Earlier quoted context omitted.

If everyone has SSH access to a server somewhere, it's easy to just put a git repo there to let everyone keep working. Obviously, this doesn't trivially scale to many repos/users/etc, which is why Github exists in the first place.

It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all. However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still…

It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.

Re: GitHub was down

#160

Earlier quoted context omitted.

At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked ? Get your popcorn out. It will happen.

I'm pretty sure most, if not all, government agencies require on-premises hosting of code

There's e.g. https://aws.amazon.com/govcloud-us/details/ these days. Azure and Google have government offerings as well. Safe to say neither would be bothering if the government didn't use cloud products these days.

http://fortune.com/2016/09/02/us-government-embraces-cloud/

Looks like even the CIA is using some AWS.

"U.S. government agencies are moving to cloud computing and away from their own data centers faster than private corporations"

Post reply on HN