Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

status.github.com

Post reply on HN
Print view View original

91–100 of 197 posts

Re: GitHub was down

#91

Post by cat199 »

brianwawok wrote: post #75
myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

You are thinking small scale, and public web.

The fact that VAXClusters were already going out of fashion by that time, and the fact of IBM's parallel sysplex existing by that time already negate your point.

previous $employer had built a globally distributed multisite c&c processing system twice by then and were in the process of revamping it for a 3rd.. 1st on mainframes & remote serial compute devices in the 80s, then on unix workstations in the early-mid 90s.

They were by far not alone in dealing with this level of complexity for the time.

Re: GitHub was down

#92

Post by myrandomcomment »

brianwawok wrote: post #75
myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

My ISP?

5 racks in 32 cities as POP. Each POP had 10 Sun Netra T1 1RU boxes behind SLB. All those were a service cache to offload the main DC which had...hum... ~100 or so Sun E450 (4x480Mhz Usparc3, 4GB RAM, 20x4GB or 9GB UltraSCSI3 drives, 2x1G NICs).

There was a cluster filesystem also. We had huge databases, etc.

This is just not that hard today.

Re: GitHub was down

#93

Post by myrandomcomment »

pvg wrote: post #90
myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

It was not a criticism so much as "well, it will be interesting to see what went wrong."

My bet is a human.

Re: GitHub was down

#94

Post by gvb »

madeofpalk wrote: post #74
tuxxy wrote: post #60

Earlier quoted context omitted.

They offer a local instance of GitHub actually.

Which I can't imagine to be any more reliable than hosted GitHub. But hey, I guess there's the comfort in know that when it goes down it's our fault!

Redundancy provides availability more than reliability.

The odds of GitHub going down and your local copy/instance simultaneously is very low so availability is high.

Reliability is how often GitHub or your local instance goes down.

Re: GitHub was down

#95

Post by rco8786 »

myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

> fairly simple

mmhmm

Re: GitHub was down

#96

Post by cat199 »

pvg wrote: post #90
myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

Is there, fundamentally?

Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either..

which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..

Re: GitHub was down

#98

Post by zbobet2012 »

anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Most items of ours that have a production dependency on GitHub actually point at a simpler (and much stupider) mirror. When commits are made pushes are mirrored to those repos. These repos can even be naively replicated since git's nature allows for this.

If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors.

Also make sure your tools don't lock on only one item.

Re: GitHub was down

#99

Post by hinkley »

myrandomcomment wrote: post #72

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

The future is here, it's just unevenly distributed.

Someone is doing all of the kinds of work necessary to verify that the servers are up, being useful, and nominally working the way the previous deployment did, but not everyone has the tools yet to do that (especially the last one).

We need a compete and complementary tool chain that everyone can use, and so far a lot of these are still business differentiators (i.e., proprietary)

Re: GitHub was down

#100

Post by Silhouette »

pcmonk wrote: post #29
anonu wrote: post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

If everyone has SSH access to a server somewhere, it's easy to just put a git repo there to let everyone keep working. Obviously, this doesn't trivially scale to many repos/users/etc, which is why Github exists in the first place.

It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all.

However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.

Post reply on HN