Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

status.github.com

Post reply on HN
Print view View original

101–110 of 197 posts

Re: GitHub was down

#101

Post by hinkley »

pvg wrote: post #90
myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

Indeed. I don't think I even heard of the C10K Problem until '03 and it was still unsolved. Now we have servers doing multiples of that much traffic on off the shelf equipment.

Re: GitHub was down

#103

Post by myrandomcomment »

cat199 wrote: post #96
pvg wrote: post #90

Earlier quoted context omitted.

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

Is there, fundamentally? Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either.. which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..

Agree. Hotmail IIRC had 9K or so servers. Yahoo had even more.

I think doing this at scale is simpler today because the networking is simpler, i.e., IP fabrics and VMs and containers. If you are building something today and want to use Layer2 I question your santiy.

Re: GitHub was down

#104

Post by twistedpair »

btown wrote: post #46
anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration…

> then some automated script will tell the cluster to pull from Github

GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.

Re: GitHub was down

#105

Post by pvg »

cat199 wrote: post #96
pvg wrote: post #90

Earlier quoted context omitted.

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

Is there, fundamentally? Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either.. which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..

Yeah I think there is. This is what wikipedia says Altavista

"As of 1998, it used 20 multi-processor machines using DEC's 64-bit Alpha processor. Together, the back-end machines had 130 GB of RAM and 500 GB of hard disk drive space, and received 13 million queries every day"

There are probably more read-only queries to gists per day than that.

Re: GitHub was down

#106

Post by gtirloni »

bananaboy wrote: post #84
gtirloni wrote: post #76

Earlier quoted context omitted.

There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.

But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.

Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.

Re: GitHub was down

#107

Post by twistedpair »

fishtoaster wrote: post #50
anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.

> github is down rarely enough

Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket.

Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).

Re: GitHub was down

#108

Post by stevegore »

btown wrote: post #46
anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration…

Could you elaborate on how Kubernetes makes it easier to deploy code changes directly to the cluster? Are you talking about the Docker/container image build?

Re: GitHub was down

#109

Post by dordoka »

EduardoBautista wrote: post #43
ben174 wrote: post #40

Earlier quoted context omitted.

Exactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.

Then what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.

You can set up another git server and webhook pull from the main one whenever someone pushes there, so you have a backup replica. Coders just need to setup two remotes for the repos and use the second one when main one is unavailable.

Re: GitHub was down

#110

Post by beager »

Woofles wrote: post #44
jdelsman wrote: post #32

Earlier quoted context omitted.

That is likely just their standard downtime message.

Only reason I brought it up because it was originally showing the all to familiar unicorn on the website, then it changed to also saying maintenance.

This seems like a pretty common response for a breaking incident for a scale app. Requests flow through to a failing system and trigger HTTP 500. Those requests may pachinko through the stack, making a variety of calls that can compound the degradation of a system weathering an unplanned failure state.

Engineers stop the bleeding by 503'ing requests at the perimeter or putting up a static maintenance page. This allows things like caches or DBs or app servers to cool off while a rollback or a revert goes out. Then, when the system is stable again, let requests flow through again (slowly, of course).

Post reply on HN