31–40 of 54 posts
Specifically you will need to accept the traffic on one interface and forward to a new local interface in order to apply the qdisc plug.
This is cool, but the above is still a pain in the neck.
When you run `/etc/init.d/nginx reload` I was under this impression this is zero-downtime. Is that not true then?
http://nginx.org/en/docs/control.html
> In order for nginx to re-read the configuration file, a HUP signal should be sent to the master process. The master process first checks the syntax validity, then tries to apply new configuration, that is, to open log files and new listen sockets. If this fails, it rolls back changes and continues to work with old configuration. If this succeeds, it starts new worker processes, and sends messages to old worker processes requesting them to shut down gracefully. Old worker processes close listen sockets and continue to service old clients. After all clients are serviced, old worker processes are shut down.
It brings up the new worker processes to handle new connections and wait for the existing connections to terminate. Key phrase is "does its best", I've seen edge cases that aren't really Nginx's fault that cause connections to be dropped. [e.g. OOM issues when it tries to bring up new workers]
My gut feeling is that this approach can be prone to silent misbehavior that they cannot detect. There's no reason not to patch HA to reload it's config and reapply it.
My favorite part was when you were rewarded for scouring the source code by finding an undocumented qdisc, that was golden. :) You mentioned that your colleague had produced a patch that did file handle passing? Is that patch available? My first thought on reading that was, how many other people have tried that very task. Im wondering if it would be helpful to HAPrpxy if your patch was made available? Thank you for s…
The only other qdisc I have much experience with is sch_netem, which emulates behaviors of a WAN (delay, loss, etc). I used it in this post [2] to conduct adversarial testing of MySQL replication (search 'tc qdisc').
[1] http://lxr.free-electrons.com/source/net/sched/
[2] http://engineeringblog.yelp.com/2014/03/mysql-replication-ne...
Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.
Earlier quoted context omitted.
Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket. No idea how that works in practice, but it seems like a sound concept? :)
Yes, that's how Unicorn works. But HAProxy isn't designed to inherit another process's current state (see this other comment: https://news.ycombinator.com/item?id=9371064 ), and it's probably quite complicated and error-prone. If you're going down that route, it's probably much easier to write the code required to simply re-read the config and apply a diff to the internal data structures.
Why not simply let DNS handle the routing and load balancing? Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.
Though the DNS system can inherently be resilient, within a DC most people only operate a single DNS server because a hierarchical domain scheme and DNS setup within DC is too cumbersome and is much less reactive to end point changes. Eg: Changing a service endpoint in an emergency takes way too long.
A single DNS server means, still there are propagation delays(local caches) and a single point of failure at critical moments(when there is a thundering herd)
Why not simply let DNS handle the routing and load balancing? Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.
My gut feeling is that this approach can be prone to silent misbehavior that they cannot detect. There's no reason not to patch HA to reload it's config and reapply it.
We're using this for internal load balancing, so we control both ends of the connection. If this started misbehaving, we'd see timeouts, dropped connections, or errors, which would show up in logs.
Earlier quoted context omitted.
That's an important detail, as it explains why the tests aren't entirely flawed: Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic. But if reloads are part of what amounts to doing live testing on a large part of live traffic -- then the exercise makes more sense. And to be sure; being able to do ten reloads every second with few ill effect…
> Testing 10 reloads every second really distorts the numbers, assuming a reload every hour, or every few hours is more realistic. It depends on what you do. I've seen shops (successfully and, IMO, correctly) scaling AWS instances for services with a threshhold of every fifteen minutes, and I've seen Mesos clusters dynamically spinning up web instances much more nimbly than that (think every two minutes under spiky l…