Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLB: GitHub's open source load balancer

githubengineering.com

Post reply on HN
Print view View original

61–65 of 65 posts

Re: GLB: GitHub's open source load balancer

#61

Post by gbrayut »

bogomipz wrote: post #60

I am having trouble understanding this passage. I'm wondering is someone could help me understand this as it seems like an important design detail: >"Another benefit to using UDP is that the source port can be filled in with a per-connection hash so that they are flow within the datacenter over different paths (where ECMP is used within the datacenter), and received on different RX queues on the proxy server’s NIC (w…

Just a guess, but by using UDP transport for the encapsulated data and configuring the module on the proxy to accept UDP on a wide range of ports, you can pick any port you want (not just the destination port of the TCP stream in the encapsulated packet). And if you are using ECMP with a known hashing algorithm you can then use that UDP port to explicitly spread packets across the RX queues on the proxy servers (gaining better performance).

Re: GLB: GitHub's open source load balancer

#62

Post by atombender »

user5994461 wrote: post #52
atombender wrote: post #28

Earlier quoted context omitted.

Round-robin routing is one way that really doesn't cut it at the level of scale that Github and others operate at. Overlay networks also typically have no notion of load and how to send traffic to the hop with the least amount of load.

Use least connections for HTTP load balancing, it is load aware.

That's an HAProxy option. What do you do to load-balance HAProxy itself? That's what the article about.

Re: GLB: GitHub's open source load balancer

#63

Post by SEJeff »

gbrayut wrote: post #55
SEJeff wrote: post #7

Earlier quoted context omitted.

Or VRRP with the open source keepalived, which has been around for a decade+ and works wonderfully on Linux.

That is exactly what Stack Overflow uses: keepalived to manage a virtual IP between two decent sized baremetal HAProxy servers (w/ bonded 10G nics). Works great and combined with DNS or Anycast based load balancing can scale pretty damn well. Definitely worth investigating as a KISS approach. To quote a recent Atwood tweet: "if I have learned anything in my career, it is the shocking effectiveness of building ... lit…

Indeed! It powers all of the internal loadbalancing (non-direct customer facing) for ticketmaster.com. I was on the core systems team ~12 or so years ago and learned all about how great it is.

Re: GLB: GitHub's open source load balancer

#64

Post by bogomipz »

gbrayut wrote: post #61
bogomipz wrote: post #60

I am having trouble understanding this passage. I'm wondering is someone could help me understand this as it seems like an important design detail: >"Another benefit to using UDP is that the source port can be filled in with a per-connection hash so that they are flow within the datacenter over different paths (where ECMP is used within the datacenter), and received on different RX queues on the proxy server’s NIC (w…

Just a guess, but by using UDP transport for the encapsulated data and configuring the module on the proxy to accept UDP on a wide range of ports, you can pick any port you want (not just the destination port of the TCP stream in the encapsulated packet). And if you are using ECMP with a known hashing algorithm you can then use that UDP port to explicitly spread packets across the RX queues on the proxy servers (gain…

Thanks I think that might be what they mean - hash the source port for better distribution across the RX queues on the destination proxy. Cheers.

Re: GLB: GitHub's open source load balancer

#65

Post by bogomipz »

The article states:

>"Each server has a bonded pair of network interfaces, and those interfaces are shared between DPDK and Linux on GLB director servers."

What's the distinction between DPDK and Linux here? It wasn't clear to me why SR-IOV is needed in this design. Does DPGK need to "own" the entire NIC device is that? In other words using DPDK and regular kernel networking are mutually exclusive option on the NIC? Is that correct?

Post reply on HN