Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

True Zero-Downtime HAProxy Reloads

engineeringblog.yelp.com

Post reply on HN
Print view View original

1–10 of 54 posts

Re: True Zero-Downtime HAProxy Reloads

#3

Post by donflamenco »

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use.

Why do you have to reload haproxy? When you update the configuration?

I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either.

Interesting hack nonetheless (stopping SYNs.)

Re: True Zero-Downtime HAProxy Reloads

#4

Post by fxx2 »

donflamenco wrote: post #3

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use. Why do you have to reload haproxy? When you update the configuration? I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either. Interesting hack nonetheless (stopping SYNs.)

Reloading HAProxy is part of the SmartStack architecture, IIRC. SmartStack/synapse does polling on say, ZooKeeper or Docker or whatever, generates a new HAProxy configuration and then reloads it.

Re: True Zero-Downtime HAProxy Reloads

#5

Post by thwarted »

pitr wrote: post #2

Does this issue exist in servers like Unicorn? A new instance forks from the old one, inheriting sockets, and starts handling requests. http://unicorn.bogomips.org/SIGNALS.html

I suspect inherited sockets across forks don't exhibit this, if a test done by Nicholas Grilly described on the haproxy mailing list [0] is to be believed (no code though, but shouldn't be hard to reproduce).

One thing that makes doing this in haproxy difficult is that there is not really any shared state between the parent and child processes, so the child doesn't have a good way to know which file descriptor maps to which listening endpoint in the configuration, since the child pretty much throws its entire state away and reads the config file anew. It's not that there can't be any shared state, but that's not how it's been architected. Finding out the endpoint via something like getsockname(2) might be doable, but the mapping of listening endpoints to listen configuration blocks isn't one-to-one, so it's actually "safer" (from an amount of code standpoint) to use SO_REUSEPORT and let the OS handling the shared listening.

[0] http://permalink.gmane.org/gmane.comp.web.haproxy/14143

Re: True Zero-Downtime HAProxy Reloads

#8

Post by Thaxll »

donflamenco wrote: post #3

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use. Why do you have to reload haproxy? When you update the configuration? I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either. Interesting hack nonetheless (stopping SYNs.)

You should read that, it's very interesting:

http://nginx.org/en/docs/control.html

Re: True Zero-Downtime HAProxy Reloads

#9

Post by fideloper »

nodesocket wrote: post #6

When you run `/etc/init.d/nginx reload` I was under this impression this is zero-downtime. Is that not true then?

Correct - article explains what HAProxy reload does behind the scenes and how it can drop connections for a very short period (many if you have many connections like yelp does!)

Re: True Zero-Downtime HAProxy Reloads

#10

Post by fideloper »

donflamenco wrote: post #3

I'm not familiar with HAProxy, but with other load balancers. I've been looking into it as a replacement for the hardware load balancers we use. Why do you have to reload haproxy? When you update the configuration? I reload nginx all the time (nginx -s reload) and I'm not sure if that is a true zero-downtime reload either. Interesting hack nonetheless (stopping SYNs.)

Right the configuration is loaded into memory, and you need to reload it to get any changes in.

I believe it's API let's you enable/disable existing configured servers but not dynamically add or remove them.

Post reply on HN