Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The GitHub Load Balancer

githubengineering.com

Post reply on HN
Print view View original

31–40 of 137 posts

Re: The GitHub Load Balancer

#31

Post by jssjr »

benologist wrote: post #25
nathankleyn wrote: post #6

Earlier quoted context omitted.

They mention in the post that they will be open-sourcing this code over the coming weeks.

They only said they will release components of it as open source. They do not say if a coherent piece of software will ever be released but today would have been the traditional time to do so for actually open source projects. It looks like marketers leveraging open source on their marketing blog that leverages developers.

A "whole" piece of software would require you to have made the same data center design decisions we've made at GitHub. While some of our choices are opinionated, I think you'll find the GLB architecture adheres to the unix philosophy of individual components each doing one thing well.

Either way, I hope the upcoming engineering-focused posts are interesting and informative! Developing GLB was a challenging engineering project and if open-sourcing it means other companies can benefit from our work and spend more time developing their products, then I'll consider that a success.

Re: The GitHub Load Balancer

#33

Post by [deleted] »

dcgudeman wrote: post #7
otoburb wrote: post #2

Given this is based on HAProxy and seems to improve the director tier of a typical L4/L7 split design, I'm led to believe GLB is an improved TCP-only load balancer. But they also talk about DNS queries, which are still mainly UDP53, so I'm hoping GLB will have UDP load-balancing capability as gravy on top. I excluded zone transfers, DNSSEC traffic or (growing) IPv6 DNS requests on TCP53 because, at least in carrier n…

Where did it say that it is based on HAProxy?

[deleted]

Re: The GitHub Load Balancer

#34

Post by bogomipz »

dcgudeman wrote: post #7
otoburb wrote: post #2

Given this is based on HAProxy and seems to improve the director tier of a typical L4/L7 split design, I'm led to believe GLB is an improved TCP-only load balancer. But they also talk about DNS queries, which are still mainly UDP53, so I'm hoping GLB will have UDP load-balancing capability as gravy on top. I excluded zone transfers, DNSSEC traffic or (growing) IPv6 DNS requests on TCP53 because, at least in carrier n…

Where did it say that it is based on HAProxy?

If you look under "stay tuned" it says:

"Now that you have a taste of the system that processed and routed the request to this blog post we hope you stay tuned for future posts describing our director design in depth, improving haproxy hot configuration reloads and how we managed to migrate to the new system without anyone noticing."

That leads me to believe it involves HAProxy.

Re: The GitHub Load Balancer

#35

Post by bogomipz »

jimjag wrote: post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

I'm not sure I agree, they mention HAProxy and Foo over UDP so they are leveraging existing open source technologies. Custom additions to suit ones particular use case isn't necessarily the same as NIH syndrome.

Re: The GitHub Load Balancer

#36

Post by RodericDay »

lifeisstillgood wrote: post #14

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms. Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.…

I started coding 3 years ago, and only started playing around with VPSs a year ago. It's hard to explain how... unaware I was, that I could just have a little "plot of land" on the internet, not managed by anyone other than me (and DO).

Re: The GitHub Load Balancer

#37

Post by jimjag »

bogomipz wrote: post #35
jimjag wrote: post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

I'm not sure I agree, they mention HAProxy and Foo over UDP so they are leveraging existing open source technologies. Custom additions to suit ones particular use case isn't necessarily the same as NIH syndrome.

They mention HAProxy, but it doesn't look like it's based on it.

Re: The GitHub Load Balancer

#38

Post by falcolas »

meddlepal wrote: post #10
tadelle wrote: post #8

Earlier quoted context omitted.

I'm talking about entire GitHub not components or Load Balancer...

Why do you think they should open source it?

My two cents? Because they have based their entire business and clientele around an open source piece of technology. Their contributions back to the community which gave them the core of their company feels... anemic.

Even if their entire stack was open sourced, GitHub would still be a profitable company simply because it's GitHub. They're hosting hundreds of thousands of repos on bare metal and providing tools around those repos allowing for the maintenance of a community - that's why they are making money; not because of some secret sauce in the code. Otherwise, GitLabs would have eaten their lunch by now.

I could definitely be wrong, and they do have 8ish pages of projects they offer up themselves, but so many are just meaningless dumps of one-off projects.

Re: The GitHub Load Balancer

#39

Post by the_duke »

cortesoft wrote: post #13
tadelle wrote: post #8

Earlier quoted context omitted.

I'm talking about entire GitHub not components or Load Balancer...

I think the model of "keep our core business software proprietary and open source all non-core software" is a good one, and one we should encourage all companies to share. If they open sourced their core site, how would they make any money?

Agreed.

Considering the amazing value Github provides for the IT world, I really don't care if they release their code.

Isn't it enought that they have revolutionized development workflows, project hosting and made contributing and collaborating so much easiert?

(to be clear, GH wasn't the first site of it's kind, but the network effect of having one site that that basically every developer uses daily -and be it just via one of the documentation site on GH pages- really was revolutionary)

Re: The GitHub Load Balancer

#40

Post by jssjr »

bogomipz wrote: post #30

Do the Directors use Anycast then? That wasn't clear to me.

Anycast usually implies traffic will be directed to the nearest node advertising that prefix. The GLB directors leverage ECMP which provides the ability to balance flows across many available paths.
Post reply on HN