Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

True Zero-Downtime HAProxy Reloads

engineeringblog.yelp.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: True Zero-Downtime HAProxy Reloads

#21

Post by click170 »

My favorite part was when you were rewarded for scouring the source code by finding an undocumented qdisc, that was golden. :)

You mentioned that your colleague had produced a patch that did file handle passing? Is that patch available? My first thought on reading that was, how many other people have tried that very task. Im wondering if it would be helpful to HAPrpxy if your patch was made available?

Thank you for sharing the plug qdisc, this is hugely useful in any number of situations. Did you find any others by chance?

I can see why some would say this solution is brittle, but actually I quite like it. It feels elegant and The Right Way to do it.

Re: True Zero-Downtime HAProxy Reloads

#23

Post by moe »

snowman17 wrote: post #20
moe wrote: post #17

Earlier quoted context omitted.

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

The solution you mention has some tradeoffs. Imagine that you have hundreds of services each of which have a port and have some long running connections. With your solution you can only restart the load balancer at the rate of the longest running connections or whenever ports run out, whichever comes first. Also then you'd have to maintain the logic to do the port mapping, iptables switching, draining, etc ... The po…

With your solution you can only restart the load balancer at the rate of the longest running connections

Well, perhaps I should have provided more detail. Our ansible deployment script actually counts the port up until it finds a free one (we keep a port-range reserved for this purpose). So if there are multiple changes in rapid succession then more than two haproxy instances may dangle around for a while.

The "port discovery" is a shell-task that registers the port to be used as a variable, which is then used by the templates of the haproxy and iptables roles.

The cleanup is done by a 15min-cronjob which kills all haproxy instances that have no connections in netstat and don't match the haproxy pidfile.

Re: True Zero-Downtime HAProxy Reloads

#25

Post by e12e »

fxx2 wrote: post #4
donflamenco wrote: post #3

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use. Why do you have to reload haproxy? When you update the configuration? I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either. Interesting hack nonetheless (stopping SYNs.)

Reloading HAProxy is part of the SmartStack architecture, IIRC. SmartStack/synapse does polling on say, ZooKeeper or Docker or whatever, generates a new HAProxy configuration and then reloads it.

That's an important detail, as it explains why the tests aren't entirely flawed: Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic. But if reloads are part of what amounts to doing live testing on a large part of live traffic -- then the exercise makes more sense.

And to be sure; being able to do ten reloads every second with few ill effects enables different, more nimble systems engineering.

But if we assume 2000 requests per second, per box - fighting ~100 reset connections a day (assuming two ha-proxy reconfigs) doesn't really seem worth the effort - packet loss and other outages would probably(?) domminate anyway.

Re: True Zero-Downtime HAProxy Reloads

#26

Post by jfroma »

donflamenco wrote: post #3

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use. Why do you have to reload haproxy? When you update the configuration? I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either. Interesting hack nonetheless (stopping SYNs.)

It is. Nginx launch a new worker while the other with the old config is still running, the master redirects new traffic to the worker and keeps the old worker until all previous requests has been handled. Once the old worker has finished the master process close it or when it time outs. It works even for updating nginx binaries.

It is a very useful approach and I use it all time as well.

I implemented the same in node by using the cluster api [1].

[1]: http://joseoncode.com/2015/01/18/reloading-node-with-no-down...

Re: True Zero-Downtime HAProxy Reloads

#27

Post by [deleted] »

moe wrote: post #17
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

[deleted]

Re: True Zero-Downtime HAProxy Reloads

#28

Post by [deleted] »

[deleted] wrote: post #27
moe wrote: post #17

Earlier quoted context omitted.

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

[deleted]

[deleted]

Re: True Zero-Downtime HAProxy Reloads

#29

Post by eropple »

e12e wrote: post #25
fxx2 wrote: post #4

Earlier quoted context omitted.

Reloading HAProxy is part of the SmartStack architecture, IIRC. SmartStack/synapse does polling on say, ZooKeeper or Docker or whatever, generates a new HAProxy configuration and then reloads it.

That's an important detail, as it explains why the tests aren't entirely flawed: Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic. But if reloads are part of what amounts to doing live testing on a large part of live traffic -- then the exercise makes more sense. And to be sure; being able to do ten reloads every second with few ill effect…

> Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic.

It depends on what you do. I've seen shops (successfully and, IMO, correctly) scaling AWS instances for services with a threshhold of every fifteen minutes, and I've seen Mesos clusters dynamically spinning up web instances much more nimbly than that (think every two minutes under spiky load--the instances would come up in five seconds, so it didn't hurt to down them).

Re: True Zero-Downtime HAProxy Reloads

#30

Post by meatmanek »

moe wrote: post #17
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

The HAProxies in question are dealing with at least one port per service, with dozens of services. If you were to remap ports, you'd end up with a lot of port mapping rules.

We also considered using different loopback IPs per HAProxy, so ports could stay consistent, but decided against it:

- We have other things (like scribe) listening on the same loopback IP address, so we'd either have to move those to different IPs or exclude them from iptables rules.

- We thought it could be confusing/misleading to see HAProxy listening on one IP/port, but connections being made to a different IP/port.

Post reply on HN