Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The GitHub Load Balancer

githubengineering.com

Post reply on HN
Print view View original

81–90 of 137 posts

Re: The GitHub Load Balancer

#81

Post by positr0n »

olalonde wrote: post #57
alexandros wrote: post #42

Earlier quoted context omitted.

here's an alternative interpretation: Due to their unique requirements, they were forced to investigate alternative approaches to the problem, and the approach they went with, they believe, is useful to a lot more people than just themselves. This is both plausible and matches the announcement, so why take a bad faith approach to someone releasing some code to the open source? Would you prefer if they kept it closed?…

> Due to their unique requirements [...] This is both plausible I'm skeptical that Github's load balancing requirements are that different from any of the other large file hosts and SaaS companies. But it's possible and I'm not in a position to tell. That being said, the "NIH syndrome" is a largely overlooked problem in our industry and I think it's reasonable to raise concerns over new projects that may be reinventi…

Some of the requirements they list are pretty unique to github since they serve .git to git clients as well as http to http clients (which is what most SaaS companies do). Like a very long running git clone from someone with slow internet not having its connection dropped.

Re: The GitHub Load Balancer

#82

Post by voltagex_ »

logicalstack wrote: post #63
NatW wrote: post #60

I'm curious if they looked into pf / CARP as part of their research into allowing horizontal scalability for an ip. See: https://www.openbsd.org/faq/pf/carp.html

CARP and similar systems require an active/passive configuration which we did not want since it needs at least twice as many hosts, half of which are not doing any work. We had similar issues with our former Git storage system based on DRDB ( http://githubengineering.com/introducing-dgit/ ). pfsync, lvs and etc uses multicast to share connection state which we also wanted to avoid.

Why do you want to avoid multicast?

Re: The GitHub Load Balancer

#83

Post by sytse »

treve wrote: post #62

I half expect a comment here explaining why Gitlab does it better ;)

:) We're not doing this better.

We're struggling with our load balancers right now. We're using Azure load balancers and then HAproxy. But the Azure ones sometimes don't work. Luckily the new network type on Azure supports floating IPs so we can set something up ourselves https://gitlab.com/gitlab-com/infrastructure/issues/466

Re: The GitHub Load Balancer

#84

Post by dpcx »

falcolas wrote: post #38
meddlepal wrote: post #10

Earlier quoted context omitted.

Why do you think they should open source it?

My two cents? Because they have based their entire business and clientele around an open source piece of technology. Their contributions back to the community which gave them the core of their company feels... anemic. Even if their entire stack was open sourced, GitHub would still be a profitable company simply because it's GitHub. They're hosting hundreds of thousands of repos on bare metal and providing tools aroun…

By that definition Google and Facebook should do the same. Or asking a farmer to give away their seeds for a similar reason.

Re: The GitHub Load Balancer

#86

Post by serge2k »

jimjag wrote: post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

> I am increasingly bothered by the "not invented here" syndrome

I'm bothered by the increasing prevalence of "never invent here".

FOSS is great, but if it's not meeting your needs then writing your own is perfectly valid.

Re: The GitHub Load Balancer

#87

Post by eikenberry »

jimjag wrote: post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

NIH syndrome is overrated. A small application doing specifically what you want is usually more robust, easier to maintain and easier extend as you need than a pre-existing application that includes most of what you need as a subset of its functionality.

Re: The GitHub Load Balancer

#88

Post by josegonzalez »

lifeisstillgood wrote: post #79
audleman wrote: post #66

Earlier quoted context omitted.

Are you saying instead of having Github, we should all be hosting our own Git repos? > But it is beginning to creak. GitHub faces scaling challenges, I don't agree that Github facing scaling issues means the web is creaking. More like old wooden boats are being replaced by big, sturdy battleships. I think the web is getting stronger thanks to engineers facing the challenges coming their way. > I am frustrated that so…

>>> Are you saying instead of having Github, we should all be hosting our own Git repos? Well, yes. That's the point. It was designed as an entirely distributed setup. It's crazy that in order to post a message to my neighbours I have to send data to Facebook in SV and just as crazy that two devs on the same team need to write their code commits in a load balanced mega server in ... Err ... Washington? Wherever. And…

No one is forcing you to use github or facebook. I host my own gitlab installation for certain private repositories and I still send emails to some people when coordinating outings.

Re: The GitHub Load Balancer

#89

Post by sanderjd »

gm-conspiracy wrote: post #44
sanderjd wrote: post #27

Earlier quoted context omitted.

I have never worked on a piece of proprietary software that would be in a state where it could be released publicly in any form without a very large outlay of development time and money. I'd be very surprised if github is any different. "Why not" is for things that are easy to do. Things that are hard and / or expensive to do require strong justification.

Honest question - do you think if the proprietary software you worked on had been planned from the beginning to be released publicly, a very large outlay would be unnecessary?

Not at all - if it were part of the strategy from the outset, different decisions would be made along the way. I wasn't trying to say that it's necessarily harder to build software for public release, just that it's hard to publicly release software that was not built that way. But I guess I don't really know for sure - I've never worked on a project like that.

Re: The GitHub Load Balancer

#90

Post by dogismycopilot »

sytse wrote: post #83
treve wrote: post #62

I half expect a comment here explaining why Gitlab does it better ;)

:) We're not doing this better. We're struggling with our load balancers right now. We're using Azure load balancers and then HAproxy. But the Azure ones sometimes don't work. Luckily the new network type on Azure supports floating IPs so we can set something up ourselves https://gitlab.com/gitlab-com/infrastructure/issues/466

I would love to see the solution. We're also desiring to run HAProxy in Azure with keepalived (even in unicast mode). The black-box "windows based" load balancer that Azure offers is quite limited.
Post reply on HN