Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

status.github.com

Post reply on HN
Print view View original

161–170 of 197 posts

Re: GitHub was down

#161

Post by Spivak »

jlebar wrote: post #157
hetman wrote: post #57

Everyone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc. Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.

> Making your organisation too dependent on a remote service can indeed be a scary prospect At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail. I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

> But I'll posit that you never actually had control, only the illusion of control.

That's a weird way of phrasing it. When you run your own services and they break you have total control and do have the power to fix it.

When you buy SaaS you are relying on someone else. You may very well have more reliability and uptime but you are nonetheless giving up control.

Re: GitHub was down

#162

Post by delias_ »

avaer wrote: post #59
jonaf wrote: post #37

Earlier quoted context omitted.

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…

Thing is, runtime dependencies on Github is something that Github is promoting and selling now ( https://github.com/marketplace/ ). Maybe this incident puts that value proposition in a more visceral context.

naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.

Re: GitHub was down

#163

Post by jopsen »

Spivak wrote: post #161
jlebar wrote: post #157

Earlier quoted context omitted.

> Making your organisation too dependent on a remote service can indeed be a scary prospect At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail. I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

> But I'll posit that you never actually had control, only the illusion of control. That's a weird way of phrasing it. When you run your own services and they break you have total control and do have the power to fix it. When you buy SaaS you are relying on someone else. You may very well have more reliability and uptime but you are nonetheless giving up control.

> When you run your own services and they break you have total control and do have the power to fix it.

Sure, but most of the time you just reboot the server or some other temporary hack and kick the problem down the road.

The power and skill to fix isn't worth much if you don't have the time.

Re: GitHub was down

#164

Post by gumby »

anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

> Run my own git server like the good 'ol days?

Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).

Re: GitHub was down

#165

Post by mygo »

mehwoot wrote: post #122
avaer wrote: post #119

Earlier quoted context omitted.

Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't. If Github being down means you're not fixing your site today, I'd call it a runtime dependency.

I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.

see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.

Re: GitHub was down

#166

Post by solatic »

ram_rar wrote: post #125
acchow wrote: post #118

Earlier quoted context omitted.

Uptime % includes time to recovery. Since you're not up if you're still recovering...

But that still does not provide much inside into recovery time. Its good to know, the median recovery time.

The only time that uptime would not give insight* into recovery time is if the service is not being regularly updated (remember, zero-downtime deployments should never affect your uptime numbers), then, the uptime numbers improve over time to something really impressive but go in the toilet once the shit hits the fan.

I hardly think it's reasonable to consider services like GitHub or infrastructure like AWS as being irregularly updated...

Re: GitHub was down

#167

Post by flukus »

jlebar wrote: post #157
hetman wrote: post #57

Everyone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc. Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.

> Making your organisation too dependent on a remote service can indeed be a scary prospect At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail. I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

Internal has always been much less reliable for me as well, the difference is it's generally unreliable outside of office hours when I don't care, hosted services often go down in the middle of my workday, which is the middle of the night in America coincidentally.

Re: GitHub was down

#168

Post by trustfundbaby »

Spivak wrote: post #161
jlebar wrote: post #157

Earlier quoted context omitted.

> Making your organisation too dependent on a remote service can indeed be a scary prospect At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail. I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

> But I'll posit that you never actually had control, only the illusion of control. That's a weird way of phrasing it. When you run your own services and they break you have total control and do have the power to fix it. When you buy SaaS you are relying on someone else. You may very well have more reliability and uptime but you are nonetheless giving up control.

That's exactly the point. Sure you have "control" but what good is all that control if it takes you 4 hours to track down the source of a problem and fix it (for example). Github has 100s of engineers (with heavily specialized knowledge that you don't have btw) working to fix any problems, I'd bet on them over myself and (maybe a handful of engineers) everytime ... and I've run my own subversion/git servers before.

Some folks just hate the feeling of not knowing what's happening and how long its going to take to fix, vs having direct access to work on problems themselves even though its not necessarily "better" in any sense of the word ... hence the illusion

Re: GitHub was down

#169

Post by trustfundbaby »

keithnz wrote: post #42
batbomb wrote: post #21

Earlier quoted context omitted.

Work locally? It's Git.

sure, but if your Build System / CI is based around github, then you are a bit poked. thank goodness I'm on bitbucket :) One of the things they offer is your own local server synched with their cloud servers for redundancy.

> sure, but if your Build System / CI is based around github, then you are a bit poked.

then architect your build system be able to switch to another service with the flip of a switch, if its that important to you.

Re: GitHub was down

#170

Post by flukus »

trustfundbaby wrote: post #169
keithnz wrote: post #42

Earlier quoted context omitted.

sure, but if your Build System / CI is based around github, then you are a bit poked. thank goodness I'm on bitbucket :) One of the things they offer is your own local server synched with their cloud servers for redundancy.

> sure, but if your Build System / CI is based around github, then you are a bit poked. then architect your build system be able to switch to another service with the flip of a switch, if its that important to you.

Github has added all sorts of proprietary integration points to prevent it from being this easy, like push based checkin notifications. Then developers get lure by the drag'n'drop CI servers that integrate with them.

Unfortunately people only care about API's now instead of standards.

Post reply on HN