Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Down with a 503

status.github.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: GitHub Down with a 503

#61

Post by Silhouette »

AYBABTME wrote: post #24
akerl_ wrote: post #13

Earlier quoted context omitted.

It's too bad git requires a central server.

How is this a useful comment? Everybody knows that Git is distributed. Everytime Github's down, somebody feels the urge to remind everybody else about that. Github's not only Git. Lots of services integrate with Github. Stuff like continuous integration, continuous deployment. If your system is built on those things, Github being down will prevent you from deploying. What's the alternative, replicate everything in-ho…

What's the alternative, replicate everything in-house/self hosted?

Erm... Yes? You write that as if hosting your own repository is somehow a foolish or implausible thing to do.

Should startup stop using third party service providers?

Stop? Maybe, maybe not, it obviously depends on the circumstances.

Stop relying on them without a back-up plan? Absolutely. If you can't run your essential systems without GitHub, or any other third party system that isn't under your direct control, then you need better contingency planning.

Re: GitHub Down with a 503

#62

Post by increment_i »

bricestacey wrote: post #51
dmoy wrote: post #6

Random somewhat on-topic question. How does one measure ones technical capability to keep up services that are more available and dependent than third party solutions? I mean obviously starting out, almost everybody can benefit from doing a lot of stuff not in-house, but at some point you'd be able to do better yourselves. But then maybe for certain services from certain providers, raw technical capability never catc…

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

But, hypothetically speaking as of now, could this be automated? And if so, how?

Re: GitHub Down with a 503

#63

Post by ereckers »

rsanheim wrote: post #54
ereckers wrote: post #2

I was waiting for this post :) Let's see who the Saturday team is!

We have an amazing team of people who are available and on-call 24/7 for these sorts of incidents.

Oh I don't doubt that. I meant finding out who was lurking around HN on a late Saturday early Sunday. The team at Github does great work!

Re: GitHub Down with a 503

#64

Post by hamburglar »

akerl_ wrote: post #30
AYBABTME wrote: post #24

Earlier quoted context omitted.

How is this a useful comment? Everybody knows that Git is distributed. Everytime Github's down, somebody feels the urge to remind everybody else about that. Github's not only Git. Lots of services integrate with Github. Stuff like continuous integration, continuous deployment. If your system is built on those things, Github being down will prevent you from deploying. What's the alternative, replicate everything in-ho…

If your workflow involves services that integrate with GitHub, you should have a plan for when GitHub is not available. You've got your source code elsewhere, you should have docs and procedures for running the tests in your own dev environments and for utilizing whatever staging/deploy process you prefer without GitHub as an intermediary. I continue to post comments like that because while I love GitHub, watching th…

100%. GitHub is amazing. GitHub is a wonderful tool. GitHub is also just a service cooked up by a handful of jokers who got enough funding to make it happen, and they do a reasonable job of keeping it running most of the time. I wish them all the best, and yet making my own company's infrastructure reliant on it is not something I'm comfortable with. Sometimes they're going to blow it, and that's ok. If them blowing it means my business is affected, I'm the dummy.

Re: GitHub Down with a 503

#65

Post by brokenparser »

akerl_ wrote: post #30
AYBABTME wrote: post #24

Earlier quoted context omitted.

How is this a useful comment? Everybody knows that Git is distributed. Everytime Github's down, somebody feels the urge to remind everybody else about that. Github's not only Git. Lots of services integrate with Github. Stuff like continuous integration, continuous deployment. If your system is built on those things, Github being down will prevent you from deploying. What's the alternative, replicate everything in-ho…

If your workflow involves services that integrate with GitHub, you should have a plan for when GitHub is not available. You've got your source code elsewhere, you should have docs and procedures for running the tests in your own dev environments and for utilizing whatever staging/deploy process you prefer without GitHub as an intermediary. I continue to post comments like that because while I love GitHub, watching th…

> especially when GitHub lives on top of a tool designed to avoid SPoFs

You're saying it as if it were a bad thing.

Re: GitHub Down with a 503

#66

Post by akerl_ »

brokenparser wrote: post #65
akerl_ wrote: post #30

Earlier quoted context omitted.

If your workflow involves services that integrate with GitHub, you should have a plan for when GitHub is not available. You've got your source code elsewhere, you should have docs and procedures for running the tests in your own dev environments and for utilizing whatever staging/deploy process you prefer without GitHub as an intermediary. I continue to post comments like that because while I love GitHub, watching th…

> especially when GitHub lives on top of a tool designed to avoid SPoFs You're saying it as if it were a bad thing.

No? The fact that GitHub is based on a distributed tool is a large part of what makes it so awesome. I can take advantage of the sweet social/collaborative elements of GitHub without locking myself to a central server. If the lower-level they used weren't so open/distributed, I doubt GitHub would have taken off like it has.

Re: GitHub Down with a 503

#68

Post by Aqueous »

plorkyeran wrote: post #60
jsmthrowaway wrote: post #42

Earlier quoted context omitted.

Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down? Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- ther…

Two third parties, even. Github's uptime is good enough that I don't see anything inherently awful about a workflow that makes development difficult when Github is down, but any production deployment process that strictly requires anything beyond a working internet connection and a working server to deploy to is insane.

Works pretty well for me. The argument you just made could have - and has - been about any other piece of infrastructure that we now work with third parties for. DNS, EMail, managed hosting, VPS hosting, off-site backups, etc, etc etc. I rely on GitHub to store my code and handle code sharing from separated networks behind NAT - why can't I rely on it for code deployment?

The days of isolating yourself from third parties are over be cause it really doesn't make sense to spend a bunch of man hours setting up and maintaining an infrastructure that GitHub - with the rare exception of tonight - specializes in and has an entire staff dedicated to keeping up and running. Third parties have become mission critical because they save money and time and are generally reliable.

If Travis CI or GitHub stop working I can do a number of things if I really need to deploy - for instance, push to production anyway, which, btw, is also hosted on a third party hosting provider. If several large services start falling like dominoes at once we have bigger problems than just whether my service is running.

Re: GitHub Down with a 503

#69

Post by Aqueous »

jsmthrowaway wrote: post #42
Aqueous wrote: post #36

Earlier quoted context omitted.

If you're using a cloud running test service like Travis CI pushing to GitHub can trigger your whole deployment process starting with running the unit and integration tests.

Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down? Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- ther…

Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS records, which are hosted on another third party's DNS servers.

Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, you didn't need to do that. If you still did I'd like to know why you did that, because in an era where even the CIA relies on third parties to accomplish mission critical tasks it doesn't seem to make a whole lot of sense.

And that trauma you are talking about - it doesn't happen. Not rarely, not ever. For years. Tonight I only happened to see that GitHub was down because I was looking at HN anyway.

Re: GitHub Down with a 503

#70

Post by copperx »

mrmondo wrote: post #41

It's amazing how often github actually goes down - 90% of the time they don't log it on their uptime / status page - it got so bad at one point I added it to our Nagios monitoring to prove a point to our devs that they shouldn't rely on external services for builds when our internal git(lab) server hasn't gone down once this year.

How does one depend on Github for builds if git is, um, distributed?
Post reply on HN