Live data from Hacker News

GitHub Down with a 503

status.github.com

81–89 of 89 posts

Re: GitHub Down with a 503

#81

Reminds me of the classic bumbling quote: "Why don't they just host GitHub where they host the status page?"

"Why don't they make the whole damn plane out of the black box?"

I'm always impressed that life support systems on star trek fail, but artificial-gravity never ceases to work.

Re: GitHub Down with a 503

#82
post #70
post #41

It's amazing how often github actually goes down - 90% of the time they don't log it on their uptime / status page - it got so bad at one point I added it to our Nagios monitoring to prove a point to our devs that they shouldn't rely on external services for builds when our internal git(lab) server hasn't gone down once this year.

How does one depend on Github for builds if git is, um, distributed?

Pull requests, commit hooks & travis integration.

Re: GitHub Down with a 503

#83

Earlier quoted context omitted.

"Why don't they make the whole damn plane out of the black box?"

I'm always impressed that life support systems on star trek fail, but artificial-gravity never ceases to work.

Probably there are a handful of gravitons or something embedded right in the floor, or there is massive inertia to the system. We tend to forget that not all solutions require active power.

Re: GitHub Down with a 503

#84
post #70
post #41

It's amazing how often github actually goes down - 90% of the time they don't log it on their uptime / status page - it got so bad at one point I added it to our Nagios monitoring to prove a point to our devs that they shouldn't rely on external services for builds when our internal git(lab) server hasn't gone down once this year.

How does one depend on Github for builds if git is, um, distributed?

go get, npm install

BTW the official standing on dependencies in Go is to actually fork all deps you need, possibly using submodules or even vendorizing by adding src/* to your project VCS.

Re: GitHub Down with a 503

#85

Earlier quoted context omitted.

By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.

I think the general consensus for running infrastructure in house isn't superior uptime but superior control. Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant. There ar…

The feeling of everything is under control...

Re: GitHub Down with a 503

#86

Earlier quoted context omitted.

I think the general consensus for running infrastructure in house isn't superior uptime but superior control. Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant. There ar…

> Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. It is fortunate, then, that all GitHub customers are in the same timezone!

Global service should have at least 3 teams (Asia, Europe, America). 4 teams would be better. They can take shifts: Asia team joins the standup meeting of Europe, then goes home. And so on.

It might not be affordable for daily works, but perfect for critical issues.

Re: GitHub Down with a 503

#87
post #83

Earlier quoted context omitted.

I'm always impressed that life support systems on star trek fail, but artificial-gravity never ceases to work.

Probably there are a handful of gravitons or something embedded right in the floor, or there is massive inertia to the system. We tend to forget that not all solutions require active power.

This does makes one wonder that if you need to make an app that has as little downtime as possible perhaps the best solution is to port as much as possible functionality client-side with client-side caching of data.

(theoretically, if I could cache every repo I visit using squid-proxy, even if github has an hour downtime I can still access the repo's I visited in the past, not perfect, but it's something!)

Re: GitHub Down with a 503

#88
post #57

Earlier quoted context omitted.

I first check their Twitter account, and then when there's nothing there about an outage (currently isn't), I come here.

https://status.github.com/ is perhaps more useful?

Generally, but I believe it's on the same network, so it may be down as well.

Re: GitHub Down with a 503

#89
post #69

Earlier quoted context omitted.

Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS re…

> Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think. Based on this…

I recognize that the costs become less straightforward the larger you get, and that a number of larger companies have to develop their own bare metal infrastructure. But I was including companies exactly like Foursquare in that handful of tech companies.

I'd still like to see an actual cost breakdown, because to me hiring a full-time staff of 5 or 10 people to run such an infrastructure, at $80,000/year or more a pop, seems like it could quickly get just as expensive or more expensive than outsourcing those costs to another organization.

There has to be some reason Netflix is able to justify placing their entire high bandwidth streaming operation on AWS, a third party that relies on full virtualization.

Post reply on HN