Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

True Zero-Downtime HAProxy Reloads

engineeringblog.yelp.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: True Zero-Downtime HAProxy Reloads

#11

Post by lobster_johnson »

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running?

It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket.

It wouldn't work for version upgrades, of course, but that's not really the main use case.

Another option could be to start up a parallel HAProxy that listens to a different port. Then use iptables to route traffic to that port instead of the old one. Not sure if this would break connections, though, or if there is a way to prevent breaking connections.

Re: True Zero-Downtime HAProxy Reloads

#12

Post by duskwuff »

lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket.

No idea how that works in practice, but it seems like a sound concept? :)

Re: True Zero-Downtime HAProxy Reloads

#13

Post by toong »

lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

I think you can use -m state --state ESTABLISHED,RELATED to match existing tcp flows.

But that builds on conntrack, you could get in trouble if this HA proxy is public facing and handles a gazillion requests ?

Re: True Zero-Downtime HAProxy Reloads

#14

Post by lobster_johnson »

toong wrote: post #13
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

I think you can use -m state --state ESTABLISHED,RELATED to match existing tcp flows. But that builds on conntrack, you could get in trouble if this HA proxy is public facing and handles a gazillion requests ?

Probably, but realistically, it's probably fine for most shops.

Re: True Zero-Downtime HAProxy Reloads

#15

Post by lobster_johnson »

duskwuff wrote: post #12
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket. No idea how that works in practice, but it seems like a sound concept? :)

Yes, that's how Unicorn works. But HAProxy isn't designed to inherit another process's current state (see this other comment: https://news.ycombinator.com/item?id=9371064), and it's probably quite complicated and error-prone.

If you're going down that route, it's probably much easier to write the code required to simply re-read the config and apply a diff to the internal data structures.

Re: True Zero-Downtime HAProxy Reloads

#17

Post by moe »

lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another option could be to start up a parallel HAProxy that listens to a different port.

That's how we do it and the script isn't even complicated.

Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it.

Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

Re: True Zero-Downtime HAProxy Reloads

#19

Post by snowman17 »

lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

There has been ongoing work in HAProxy for many years to support zero downtime reloads via a few different potential mechanisms (file descriptor passing, a socket server, etc ...). Unfortunately it turns out that it is really hard given the architecture of HAProxy. That being said, I'm sure patches are always welcome.

The post does mention that they did consider something similar to what you are suggesting with the multiple HAProxy instances but decided against it due to engineering uncertainties. Could be that they just overestimated how hard it would be.

Re: True Zero-Downtime HAProxy Reloads

#20

Post by snowman17 »

moe wrote: post #17
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

The solution you mention has some tradeoffs. Imagine that you have hundreds of services each of which have a port and have some long running connections. With your solution you can only restart the load balancer at the rate of the longest running connections or whenever ports run out, whichever comes first.

Also then you'd have to maintain the logic to do the port mapping, iptables switching, draining, etc ...

The post mentions they considered this option and they claim that they were worried about engineering risk; seems reasonable to be worried about that.

Post reply on HN