True Zero-Downtime HAProxy Reloads • Hacker News Board

Re: True Zero-Downtime HAProxy Reloads

#51

Post by jolynch » Tue, Apr 14, 2015, 6:23 PM UTC

click170 wrote: post #21

My favorite part was when you were rewarded for scouring the source code by finding an undocumented qdisc, that was golden. :) You mentioned that your colleague had produced a patch that did file handle passing? Is that patch available? My first thought on reading that was, how many other people have tried that very task. Im wondering if it would be helpful to HAPrpxy if your patch was made available? Thank you for s…

Josh gets all the credit for finding the plug qdisc. I was originally planning on using netem and having a fixed delay of 50ms or something, but Josh decided that wasn't elegant enough and went looking for a better solution.

As for the patch, I can certainly ask John but I do know that the proof of concept was written during a hackathon so I imagine there could either be fundamental flaws or just need a lot of work to get ready to merge. From reading the HAProxy mailing list I think they have been working on this for a while, but the issue seems to be merge risk (it's a fairly large architecture change). I'll mention it to John though.

I'm glad you think it's elegant :-) I tried hard to find a solution that was minimally invasive (no code changes, no significant infrastructure, etc ...).

Re: True Zero-Downtime HAProxy Reloads

#52

Post by jolynch » Tue, Apr 14, 2015, 6:24 PM UTC

jpgvm wrote: post #31

Everyone seems to have overlooked the fact this only works for outgoing traffic, you can't use this strategy on a public facing load-balancer without additional work. Specifically you will need to accept the traffic on one interface and forward to a new local interface in order to apply the qdisc plug. This is cool, but the above is still a pain in the neck.

Totally agree, I mention this as the largest drawback of this approach in the post.

Re: True Zero-Downtime HAProxy Reloads

#53

Post by jolynch » Tue, Apr 14, 2015, 6:31 PM UTC

ksri wrote: post #35

Why not simply let DNS handle the routing and load balancing? Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.

Heh, the number of times DNS has bitten me in production... My main objections to DNS are:

1) You have no control over clients, there is absolutely no rule that says clients have to respect TTLs (I'm looking at you Java)

2) We are talking about hundreds of HAProxy load balancers, thousands of clients and hundreds of backend services which are moving around all the time. I just honestly didn't want to deal with DNS propagation limiting my flexibility.

3) At least at Yelp, we don't really have particularly nice control apis for DNS. This is sort of specific to Yelp, but it was a factor.

Re: True Zero-Downtime HAProxy Reloads

#54

Post by jolynch » Tue, Apr 14, 2015, 10:40 PM UTC

halayli wrote: post #39

meatmanek wrote: post #33

Earlier quoted context omitted.

We're using this for internal load balancing, so we control both ends of the connection. If this started misbehaving, we'd see timeouts, dropped connections, or errors, which would show up in logs.

I see. Any reason not to patch HA to reload config?

Patching HAProxy to reload config is really hard. There have been ideas, patches, and discussions on the HAProxy mailing list for a few years now trying to get zero downtime reloads natively supported in HAProxy, but the reality is that it just is not as easy as it might seem.

For more details check out the mailing list: http://marc.info/?l=haproxy