Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLB: GitHub's open source load balancer

githubengineering.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: GLB: GitHub's open source load balancer

#41

Post by tehnerd »

lmb wrote: post #11
justinsaccount wrote: post #5

See also: Facebook: https://code.fb.com/open-source/open-sourcing-katran-a-scala... Google: https://cloudplatform.googleblog.com/2016/03/Google-shares-s... The design of all 3 is very similar.

I haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies) In terms of technology, Katran uses XDP and IPIP tunnels, both upstream in the Linux kernel. GLB uses DPDK, which allows processing raw packets in user space, and Geberic UDP encapsulation + a custom iptables module. Neither DPDK nor the module are upstream. There are archit…

katran has example of how to use this library. which is (example) more like ipvsadm + kernel.

Re: GLB: GitHub's open source load balancer

#42

Post by vbernat »

llama052 wrote: post #24

Maybe I don't see the use case since I'm not at that scale, but it seems like a lot of added complexity for what appears to be hacking around using other load balancing solutions as a Layer-4 option? Edit: It's a question, if you downvote please let me know why it's a better solution.

The goal is to avoid disruption during topology changes. If you have long-lived connections, this is important to keep them alive. The explain this a bit more here: https://github.com/github/glb-director/blob/master/docs/deve...

I have also written about this in a past article: https://vincent.bernat.im/en/blog/2018-multi-tier-loadbalanc... (which may or may not be easier to understand)

Re: GLB: GitHub's open source load balancer

#43

Post by vbernat »

q3k wrote: post #17

This basically looks like an open source Maglev [1]. Awesome! [1] - https://static.googleusercontent.com/media/research.google.c...

This is similar but not totally. Linux has an open source version of Maglev as a scheduler for LVS since 4.18 (to be released). GLB uses a specific consistent hashing algorithm selecting two servers and a module to let the first server redirect the flow to the second in case it doesn't know about it. This helps minimize disruption even more than with Maglev.

Re: GLB: GitHub's open source load balancer

#44

Post by atombender »

llama052 wrote: post #30
atombender wrote: post #27

Earlier quoted context omitted.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

For the cost of a ton of added complexity though? What do you get out of this solution that other solutions don't provide? Say DNS load-balancing, VRRP, CARP, or any other HA solution.

Not for most companies, but Github's scalability requirements are pretty extensive, and really cry out for something more sophisticated than the technologies you mention.

This stuff isn't exactly new; it's essentially the Maglev system described by Google in a 2016 paper. Other companies are now catching up to Google (which is of course 2+ years ahead).

Re: GLB: GitHub's open source load balancer

#47

Post by merb »

llama052 wrote: post #30
atombender wrote: post #27

Earlier quoted context omitted.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

For the cost of a ton of added complexity though? What do you get out of this solution that other solutions don't provide? Say DNS load-balancing, VRRP, CARP, or any other HA solution.

well you can't have 100% redudancy without a virtual ip or bgp. so basically glb-director is the same as just using haproxy + bgp. (bgp can basically do anycast/ecmp multipath really easily. well you still need redudant network routers.)

Re: GLB: GitHub's open source load balancer

#48

Post by merb »

KenanSulayman wrote: post #31
atombender wrote: post #27

Earlier quoted context omitted.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

Wouldn't it be possible to use DNS for this, with multiple A entries per LB, a TTL of 30 or 60? And remove unhealthy servers from the list? That would even come with IPv6 support. Then you could address the LB with an address like some-service.lb.intranet and just use that where ever you would use the original service.

DNS failover looks like a neat idea, but does not work well that good. Until a new DNS entry propagates it could take a really really long time. also using anycast/ecmp via bgp means that you have a single ip that is highly redudant because it can be backed by many servers.

Re: GLB: GitHub's open source load balancer

#49

Post by theojulienne »

emmericp wrote: post #33

Cool use of SR-IOV, I like it. We've done a few (academic) experiments with SR-IOV for flow bifurcation and we've wondered why no one seems to use it like this. The performance was quite good: neglible performance difference between PF and a single VF and only 5-10% when running multiple >= 8 VFs (probably cache contention somewhere in our specific setup). You seem to be running this on X540 NICs, aren't you running…

We found that we could achieve 10G line rate with just the queues available to the VF, the NIC didn't seem to be a bottleneck providing DPDK was processing packets faster than line rate. It's worth noting that other traffic on the PF was/is minimal in our setup.

We tested this using DPDK pktgen on a identically-configured node (GLB Director and pktgen both using DPDK on a VF with flow bifurcation, on 2 separate machines on the same rack/switch), with GLB Director essentially acting as a reflector back to the pktgen node. pktgen was able to generate enough 40 byte TCP packets to saturate 10G with 2 TX cores/queues, and GLB Director was able to process those packets and encapsulate them with a sizeable set of binds/tables with 3 cores doing work (encapsulation) and 1 core doing RX/distribution/TX.

Re: GLB: GitHub's open source load balancer

#50

Post by user5994461 »

blantonl wrote: post #35
KenanSulayman wrote: post #31

Earlier quoted context omitted.

Wouldn't it be possible to use DNS for this, with multiple A entries per LB, a TTL of 30 or 60? And remove unhealthy servers from the list? That would even come with IPv6 support. Then you could address the LB with an address like some-service.lb.intranet and just use that where ever you would use the original service.

Problem here is you assume that every client honors the TTL. That is a very bad assumption to make.

DNS failover works well in practice.
Post reply on HN