Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

True Zero-Downtime HAProxy Reloads

engineeringblog.yelp.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: True Zero-Downtime HAProxy Reloads

#41

Post by nierman »

ksri wrote: post #35

Why not simply let DNS handle the routing and load balancing? Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.

Don't muck with the DNS record. Use an "A record" associated with a secondary/floating IP; then simply move the IP associated with the A record to the new machine and issue a gratuitous ARP.

You can avoid setting arbitrarily low--but not low enough--TTL, etc.

Re: True Zero-Downtime HAProxy Reloads

#42

Post by meatmanek »

halayli wrote: post #39
meatmanek wrote: post #33

Earlier quoted context omitted.

We're using this for internal load balancing, so we control both ends of the connection. If this started misbehaving, we'd see timeouts, dropped connections, or errors, which would show up in logs.

I see. Any reason not to patch HA to reload config?

Not sure; I'm not the author of the post (a colleague), but I'd guess it's a fairly complicated patch. I know the author has made some changes to the HAProxy codebase[1], so I'm sure he considered it.

[1] http://comments.gmane.org/gmane.comp.web.haproxy/21025

Re: True Zero-Downtime HAProxy Reloads

#43

Post by toong »

lobster_johnson wrote: post #14
toong wrote: post #13

Earlier quoted context omitted.

I think you can use -m state --state ESTABLISHED,RELATED to match existing tcp flows. But that builds on conntrack, you could get in trouble if this HA proxy is public facing and handles a gazillion requests ?

Probably, but realistically, it's probably fine for most shops.

Most shops don't handle a gazillion requests. And if you don't handle a gazillion requests, you don't really care about 20ms downtime while HA proxy is reloading ?

Re: True Zero-Downtime HAProxy Reloads

#44

Post by wereHamster »

How about using IPVS? To reload HAProxy, start a new instance, add as a new IPVS real server, set the weight of the old HAProxy to zero, wait until the connection count drops to zero, and then you can remove the old HAProxy real server.

IPVS also allows to gradually shift traffic from one instance to the other. So if you see elevated error reports from the new server, you can quickly drop it and go back to the old setup.

Re: True Zero-Downtime HAProxy Reloads

#45

Post by ekimekim »

duskwuff wrote: post #12
lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket. No idea how that works in practice, but it seems like a sound concept? :)

A less fragile approach would be for the old process to simply pass the listen fd to the new process over a unix socket. You could also pass any active connections in this way.

Re: True Zero-Downtime HAProxy Reloads

#46

Post by hueving »

mrphoebs wrote: post #37
ksri wrote: post #35

Why not simply let DNS handle the routing and load balancing? Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.

DNS is a fragile system to use within a DC to do service/endpoint discovery. That's because DNS tends to be a single point of failure. Though the DNS system can inherently be resilient, within a DC most people only operate a single DNS server because a hierarchical domain scheme and DNS setup within DC is too cumbersome and is much less reactive to end point changes. Eg: Changing a service endpoint in an emergency ta…

That's a pretty weak argument. It's like calling a databases a single point of failure because the people who set them up tend to be too lazy to setup HA.

Re: True Zero-Downtime HAProxy Reloads

#47

Post by mobiplayer »

I'm afraid I'm also of the feeling that this is too complex, especially moving forward with the business. It was probably fun to work on it and solved your issue, but somehow it looks like a patch for an architecture issue.

Does this add extra complexity to platform (software) updates?

Re: True Zero-Downtime HAProxy Reloads

#48

Post by eropple »

e12e wrote: post #40
eropple wrote: post #29

Earlier quoted context omitted.

> Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic. It depends on what you do. I've seen shops (successfully and, IMO, correctly) scaling AWS instances for services with a threshhold of every fifteen minutes, and I've seen Mesos clusters dynamically spinning up web instances much more nimbly than that (think every two minutes under spiky l…

Well, once every 120 seconds is still quite a leap from 10 every second...

Sure, but if it doesn't work there, I don't trust it to work if, say, a piece of my scheduler goes nuts and suddenly is upping and downing containers every few seconds. The problem remains, it's just not as acute and still must be fixed.

Re: True Zero-Downtime HAProxy Reloads

#49

Post by jolynch »

lobster_johnson wrote: post #11

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another optio…

This was more complicated that I hoped it would be

I also hoped that it would be less complicated, but after prototyping a few different options, many of which I mention in the blog post (e.g. a patch to do fd passing, huptime, or running multiple parallel HAProxies), I decided that this was the lowest risk given the engineering constraints that we had. Also I wasn't entirely sure of an implementation plan for how to cleanly do port mappings for the hundreds of services. For example our service provisioning system adds, change, move, or removes services in various Yelp environments dynamically, so the port mapping logic would have to keep sync with that. It is certainly possible but I stand by my concern.

To put this in perspective, I got this working in production at Yelp in about two or three days, and it has not had a single issue since. With this solution we apply some TC rules and some iptables rules once, and then the act of restarting is just a set of TC commands. It solves all of our use cases and does so without significant infrastructure that someone else has to maintain two years from now.

If a month from now HAProxy releases native zero downtime restarts, I don't feel bad about throwing all of this away.

Re: True Zero-Downtime HAProxy Reloads

#50

Post by jolynch »

wereHamster wrote: post #44

How about using IPVS? To reload HAProxy, start a new instance, add as a new IPVS real server, set the weight of the old HAProxy to zero, wait until the connection count drops to zero, and then you can remove the old HAProxy real server. IPVS also allows to gradually shift traffic from one instance to the other. So if you see elevated error reports from the new server, you can quickly drop it and go back to the old se…

I am not familiar with IPVS, I'll look into it.

From your description though I imagine it might have the same drawbacks that exist with the multiple HAProxy + iptables swap solution. With a solution where we run multiple HAProxies and move traffic between them I worry about port exhaustion, long running connections, and the maintenance costs of the infrastructure. It could just be the case that we're talking about different problems. At Yelp we have hundreds of services that are added, changed, moved, and removed from physical machines causing our internal HAProxies (which listen on hundreds of ports and many have long running connections) to reload pretty constantly.

As all engineering decisions, I could very easily be wrong and be overestimating the complexity involved with the multiple HAProxy instance solution. At the end of the day I made a call based on the data available, and I decided the solution I talked about in the blog post was lowest risk.

Post reply on HN