Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Down with a 503

status.github.com

Post reply on HN
Print view View original

71–80 of 89 posts

Re: GitHub Down with a 503

#71

Post by opendais »

bricestacey wrote: post #51
dmoy wrote: post #6

Random somewhat on-topic question. How does one measure ones technical capability to keep up services that are more available and dependent than third party solutions? I mean obviously starting out, almost everybody can benefit from doing a lot of stuff not in-house, but at some point you'd be able to do better yourselves. But then maybe for certain services from certain providers, raw technical capability never catc…

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

I think the general consensus for running infrastructure in house isn't superior uptime but superior control.

Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant.

There are very valid reasons for both choices.

Re: GitHub Down with a 503

#72

Post by __david__ »

increment_i wrote: post #62
bricestacey wrote: post #51

Earlier quoted context omitted.

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

But, hypothetically speaking as of now, could this be automated? And if so, how?

Redundancy. Lots of it. A simple example would be hitting your website with a script every so often and when it stops responding then update your DNS to point to another data center that's up and ready to go.

Re: GitHub Down with a 503

#73

Post by skrebbel »

opendais wrote: post #71
bricestacey wrote: post #51

Earlier quoted context omitted.

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

I think the general consensus for running infrastructure in house isn't superior uptime but superior control. Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant. There ar…

> Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working.

It is fortunate, then, that all GitHub customers are in the same timezone!

Re: GitHub Down with a 503

#74

Post by skrebbel »

cclogg wrote: post #9

Haha, I wonder how many of us came here because we couldn't push or pull commits, so just taking a HN reading break... and boom the top post is about Github!

Yeah maybe we should switch to a distributed VCS so that we don't have this problem anymore.

Re: GitHub Down with a 503

#75

Post by bricestacey »

increment_i wrote: post #62
bricestacey wrote: post #51

Earlier quoted context omitted.

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

But, hypothetically speaking as of now, could this be automated? And if so, how?

In short, no, unless you think people replacing broken hardware or patching bugs in your software can be automated.

Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.

If you're just interested in the subject, research high availability.

Re: GitHub Down with a 503

#76

Post by bricestacey »

increment_i wrote: post #62
bricestacey wrote: post #51

Earlier quoted context omitted.

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

But, hypothetically speaking as of now, could this be automated? And if so, how?

In short, no, unless you think people replacing broken hardware or patching bugs in your software can be automated.

Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.

If you're just interested in the subject, research high availability.

Re: GitHub Down with a 503

#77

Post by skrebbel »

hamburglar wrote: post #64
akerl_ wrote: post #30

Earlier quoted context omitted.

If your workflow involves services that integrate with GitHub, you should have a plan for when GitHub is not available. You've got your source code elsewhere, you should have docs and procedures for running the tests in your own dev environments and for utilizing whatever staging/deploy process you prefer without GitHub as an intermediary. I continue to post comments like that because while I love GitHub, watching th…

100%. GitHub is amazing. GitHub is a wonderful tool. GitHub is also just a service cooked up by a handful of jokers who got enough funding to make it happen, and they do a reasonable job of keeping it running most of the time. I wish them all the best, and yet making my own company's infrastructure reliant on it is not something I'm comfortable with. Sometimes they're going to blow it, and that's ok. If them blowing…

> jokers who got enough funding to make it happen

While I agree with the general gist of your comment, GitHub was entirely self-funded up until about a year ago or so. they didn't "get" funding, they made real money and reinvested it and grew.

Re: GitHub Down with a 503

#78

Post by hamburglar »

skrebbel wrote: post #77
hamburglar wrote: post #64

Earlier quoted context omitted.

100%. GitHub is amazing. GitHub is a wonderful tool. GitHub is also just a service cooked up by a handful of jokers who got enough funding to make it happen, and they do a reasonable job of keeping it running most of the time. I wish them all the best, and yet making my own company's infrastructure reliant on it is not something I'm comfortable with. Sometimes they're going to blow it, and that's ok. If them blowing…

> jokers who got enough funding to make it happen While I agree with the general gist of your comment, GitHub was entirely self-funded up until about a year ago or so. they didn't "get" funding, they made real money and reinvested it and grew.

Fine, bootstrapping jokers are my favorite kind of jokers. No offense intended. My point is it's just a small company with no SLA, express or implied. It could have been anybody with the same great idea, and just because they had a great idea doesn't make them automatically reliable enough to pin your infrastructure on.

Re: GitHub Down with a 503

#79

Post by jsmthrowaway »

Aqueous wrote: post #69
jsmthrowaway wrote: post #42

Earlier quoted context omitted.

Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down? Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- ther…

Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS re…

> Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies,

The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think.

Based on this comment and the recent reply in this thread I can tell you've mostly worked with small-scale architectures. Third party tooling and workflows do make a lot of sense at a small scale, but the point at which you outgrow those solutions comes a lot sooner than you think. When I arrived at Foursquare the entire operation was on Amazon; when I left a year and a half later, much of it was on physical equipment. And Foursquare is not a Google-scale operation -- virtualization and customer cotenancy just have a serious impact on SLA that is less pronounced at smaller scales.

It's easy to think your experiences represent the industry, as your comments suggest. It's also easy to think HN represents the industry, where startups reign supreme and everybody loves working Lean Devops. The fact is, neither of those statements are accurate, and beyond a six (or maybe seven) figure architecture you start having a harder time justifying third parties financially and operationally.

I do use Amazon currently, just as an off-site backup for on-site monitoring that I've built. That's common.

Re: GitHub Down with a 503

#80

Post by robbles »

eli wrote: post #56
robbles wrote: post #37

Could Github store mirrors of all repositories on some third-party service, like S3? Even if they were only updated every 24 hours and had a limited history for each repo, it still seems like it would be a really useful fallback that they could put up when things like this happen.

It's actually not that hard to set up your git to push to both Github and another repository at the same time.

Sure, it's easy to do with your own code. But what about apps that don't vendor their dependencies? If you need to pull in third-party dependencies when you deploy or test, it's still possible to maintain your own mirror, but it would be nice if you could depend on the code being "up".
Post reply on HN