Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLB: GitHub's open source load balancer

githubengineering.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: GLB: GitHub's open source load balancer

#31

Post by KenanSulayman »

atombender wrote: post #27
KenanSulayman wrote: post #25

Earlier quoted context omitted.

I have very successfully in the past and still do use HAProxy as level 4 LB. It's one of the fastest to my knowledge. I have used HAProxy as entry to big Mesos clusters without any issue before. One example of using HAProxy as L4 LB instead of letting it do the termination is when it is proxying TLS traffic from and to multiple backends. Or Websocket. Or even as bastion LB for SSH should one bastion go down.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

Wouldn't it be possible to use DNS for this, with multiple A entries per LB, a TTL of 30 or 60? And remove unhealthy servers from the list? That would even come with IPv6 support.

Then you could address the LB with an address like some-service.lb.intranet and just use that where ever you would use the original service.

Re: GLB: GitHub's open source load balancer

#32

Post by justinsaccount »

lmb wrote: post #21
justinsaccount wrote: post #16

Earlier quoted context omitted.

> uses connection tracking at the load balancer to know where to send packets for established flows Kind of, from katran post: > Each L4LB also stores the backend choice for each 5-tuple as a lookup table to avoid duplicate computation of the hash on future packets. This state is a pure optimization and is not necessary for correctness > .. > Katran uses an extended version of the Maglev hash to select the backend se…

If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.

I haven't looked at the code, but this is what they say:

> Katran uses an extended version of the Maglev hash to select the backend server. A few features of the extended hash are resilience to backend server failures, more uniform distribution of load, and the ability to set unequal weights for different backend servers.

GLB also does the same thing with caching:

> The hashing to choose the primary/secondary server is done once, up front, and is stored in a lookup table, and so doesn’t need to be recalculated on a per-flow or per-packet basis.

GLB has that primary/secondary thing which seems to be how it better handles backends coming and going.

Re: GLB: GitHub's open source load balancer

#33

Post by emmericp »

Cool use of SR-IOV, I like it. We've done a few (academic) experiments with SR-IOV for flow bifurcation and we've wondered why no one seems to use it like this. The performance was quite good: neglible performance difference between PF and a single VF and only 5-10% when running multiple >= 8 VFs (probably cache contention somewhere in our specific setup).

You seem to be running this on X540 NICs, aren't you running into limitations for the VFs. Mostly the number of queues which I believe is limited to 2 per VF in the ixgbe family. I wonder whether the AF_XDP DPDK driver could be used instead if SR-IOV isn't available or feasible for some reason.

A more detailed look at performance would have been cool. I might try it myself if I find some time (or a student) :)

Re: GLB: GitHub's open source load balancer

#35

Post by blantonl »

KenanSulayman wrote: post #31
atombender wrote: post #27

Earlier quoted context omitted.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

Wouldn't it be possible to use DNS for this, with multiple A entries per LB, a TTL of 30 or 60? And remove unhealthy servers from the list? That would even come with IPv6 support. Then you could address the LB with an address like some-service.lb.intranet and just use that where ever you would use the original service.

Problem here is you assume that every client honors the TTL. That is a very bad assumption to make.

Re: GLB: GitHub's open source load balancer

#36

Post by lmb »

justinsaccount wrote: post #32
lmb wrote: post #21

Earlier quoted context omitted.

If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.

I haven't looked at the code, but this is what they say: > Katran uses an extended version of the Maglev hash to select the backend server. A few features of the extended hash are resilience to backend server failures, more uniform distribution of load, and the ability to set unequal weights for different backend servers. GLB also does the same thing with caching: > The hashing to choose the primary/secondary server…

Ok, I've looked at the code, and based on that I think what I said in my original post is correct.

GLB is more complex, but will maintain connections in more circumstances.

Re: GLB: GitHub's open source load balancer

#38

Post by buzer »

KenanSulayman wrote: post #31
atombender wrote: post #27

Earlier quoted context omitted.

It's not that HAProxy doesn't do L4. As I said, projects like GLB solves how to make the load balancer itself redundant; how to load balance the load balancer, so to speak.

Wouldn't it be possible to use DNS for this, with multiple A entries per LB, a TTL of 30 or 60? And remove unhealthy servers from the list? That would even come with IPv6 support. Then you could address the LB with an address like some-service.lb.intranet and just use that where ever you would use the original service.

Designs such as GLB can (I haven't looked deeply enough in GLB specifically to see if they can do it or not, but I would assume so) handle director level failures mid-flow, i.e. connection won't be interrupted even if one them dies (packet losses are still likely, but TCP will take care of them). That allows a lot faster recovery than solutions that depend on client's DNS settings.

Additionally DNS will leave your load balancing at the mercy of ISPs DNS server settings. At least in the past it wasn't exactly unheard of that ISPs only cached single A entry so all of their clients would be directed to single server.

That said, DNS based load balancing is generally good enough solution for most of people.

Re: GLB: GitHub's open source load balancer

#39

Post by tehnerd »

lmb wrote: post #21
justinsaccount wrote: post #16

Earlier quoted context omitted.

> uses connection tracking at the load balancer to know where to send packets for established flows Kind of, from katran post: > Each L4LB also stores the backend choice for each 5-tuple as a lookup table to avoid duplicate computation of the hash on future packets. This state is a pure optimization and is not necessary for correctness > .. > Katran uses an extended version of the Maglev hash to select the backend se…

If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.

connection tracking could be turned off. but yeah, but default it uses it (for cases where you need to drain a host w/o distruption of existing connections). it's easy to make katran behave the same way as glb (by disabling connection tracking + writing host module which is doing more or less the same as what glb's agent is doing)

Re: GLB: GitHub's open source load balancer

#40

Post by tehnerd »

notyourday wrote: post #29
lmb wrote: post #11

Earlier quoted context omitted.

I haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies) In terms of technology, Katran uses XDP and IPIP tunnels, both upstream in the Linux kernel. GLB uses DPDK, which allows processing raw packets in user space, and Geberic UDP encapsulation + a custom iptables module. Neither DPDK nor the module are upstream. There are archit…

It is unfortunate that Fastly did not open source faild. That's the most elegant solution to draining that I have ever seen.

there is no big difference compare to what glb is doing. they are encoding next server in mac. glb is doign the same w/ metadata inside gue
Post reply on HN