Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The GitHub Load Balancer

githubengineering.com

Post reply on HN
Print view View original

131–137 of 137 posts

Re: The GitHub Load Balancer

#131

Post by jerkstate »

logicalstack wrote: post #109
bogomipz wrote: post #100

Earlier quoted context omitted.

Are you running Quagga or Bird on the director instances then? I'm looking forward to reading more about it.

We use Quagga.

This is really cool work, I worked with a team that implemented an ECMP hashing scheme using a set of IPs kept alive by VRRP in a previous lifetime, so I have a bit of familiarity with the space and a few questions.

The article says the L4 layer uses ECMP with consistent/rendezvous hashing. is this vendor implemented or implemented by you using openflow or something similar? How does graceful removal at the director layer work? I know you would have to start directing incoming SYNs to another group, but how do you differentiate non-SYN packets that started on the draining group vs. ones that started on the new group?

If you are using L4 fields in the hash, how do you handle ICMP? This approach could break PMTU discovery because a icmp fragmentation needed packet sent in response to a message sent to one of your DSR boxes might hash to a different box, unless considerations have been made.

Re: The GitHub Load Balancer

#132

Post by lamontcg »

Why not just use DNS load balancing over VIPs served by HA pairs of load balancers?

Back in the day we did this with Netscalers doing L7 load balancing in clusters, and then Cisco Distributed Directors doing DNS load balancing across those clusters.

It can take days/weeks to bleed off connections from a VIP that is in the DNS load balancing, but since you've got an H/A pair of load balancers on every VIP you can fail over and fail back across each pair to do routine maintenance.

That worked acceptably for a company with a $10B stock valuation at the time.

Re: The GitHub Load Balancer

#133

Post by jdc0589 »

sytse wrote: post #83
treve wrote: post #62

I half expect a comment here explaining why Gitlab does it better ;)

:) We're not doing this better. We're struggling with our load balancers right now. We're using Azure load balancers and then HAproxy. But the Azure ones sometimes don't work. Luckily the new network type on Azure supports floating IPs so we can set something up ourselves https://gitlab.com/gitlab-com/infrastructure/issues/466

do you guys have a blog post or anything talking about why you settled on Azure? I've used both azure and aws a decent amount now, and GitLab falls outside the "right place for the right tool" mentality I've been operating with. Most of our infrastructure is in Azure, but it's also mainly windows (azure has the best windows VM pricing with a moderate was agreement, which isn't surprising). There are some regulatory considerations for us too, buts that's anither conversation.

Re: The GitHub Load Balancer

#134

Post by manigandham »

lamontcg wrote: post #132

Why not just use DNS load balancing over VIPs served by HA pairs of load balancers? Back in the day we did this with Netscalers doing L7 load balancing in clusters, and then Cisco Distributed Directors doing DNS load balancing across those clusters. It can take days/weeks to bleed off connections from a VIP that is in the DNS load balancing, but since you've got an H/A pair of load balancers on every VIP you can fail…

Company stock value has nothing to do with their scaling, performance and customized processing requirements.

Re: The GitHub Load Balancer

#135

Post by JdeBP »

otterley wrote: post #121
logicalstack wrote: post #45

Earlier quoted context omitted.

Joe from GitHub here, frankly there is a lot we want to talk about and release and it was simply too much for one post. We'd like to give it a proper treatment and single very long post won't do that. Also, it allows us to get folks interested in the project and give us time to prepare our code for release. It's a surprisingly big job.

Personally, I would have preferred you waited until you could release all the documents at once. I admit I was interested, but I've seen too many people and organizations start a conversation but never finish it or show the goods. It's misleading and unfair to dangle a solution when all you really have is a problem. Take, for example, this post from CoreOS back in March 2016 that suggested that they might know a way…

As if by magic, part 2 has just appeared. (-: See https://news.ycombinator.com/item?id=12603322 .

Re: The GitHub Load Balancer

#136

Post by wtarreau »

Did people really read the article ? For me it was pretty clear, maybe it involves some regular load-balancing terms that people are not familiar with, because I'm seeing a lot of bullshit written in the comments, but here is what is described there :

- in a traditional L4/L7 load balancing setup (typically what is described in my very old white paper "making applications scalable with load balancing"), the first layer (L3-4 only, stateless or stateful) is often called the "director".

- the second level (L7) necessarily is based on a proxy.

For the director part, LVS used to be used a lot over the last decade, but over the last 3-4 years we're seeing ECMP implemented almost in every router and L3 switch, offering approximately the same benefits without adding machines.

ECMP has some drawbacks (breaks all connections during maintenance due to stateless hashing).

LVS has other drawbacks (requires synchronization, cannot learn previous sessions upon restart, sensitivity to SYN floods).

Basically what they did is something between the two for the director, involving consistent hashing to avoid having to deal with connection synchronization without breaking connections during maintenance periods.

This way they can hack on their L7 layer (HAProxy) without anyone ever noticing because the L4 layer redistributes the traffic targeting stopped nodes, and only these ones.

Thus the new setups is now user->GLB->HAProxy->servers.

And I'm very glad to see that people finally attacked the limitations everyone has been suffering from at the director layer, so good job guys!

Re: The GitHub Load Balancer

#137

Post by [deleted] »

manigandham wrote: post #134
lamontcg wrote: post #132

Why not just use DNS load balancing over VIPs served by HA pairs of load balancers? Back in the day we did this with Netscalers doing L7 load balancing in clusters, and then Cisco Distributed Directors doing DNS load balancing across those clusters. It can take days/weeks to bleed off connections from a VIP that is in the DNS load balancing, but since you've got an H/A pair of load balancers on every VIP you can fail…

Company stock value has nothing to do with their scaling, performance and customized processing requirements.

[deleted]
Post reply on HN