Ask HN: How do I failover the load balancer

Does this (http://www.howtoforge.com/high-availability-load-balancer-ha...) guide answer your question? Failover of the load balancer is described in sections 1 and 5 (on page 2).

The load balancers share one public IP address. Only one server at a time is assigned that address. To failover: 1. Physically isolate the failed load balancer by removing power and/or disabling its Ethernet port on the switch. 2. Assign the shared public IP address to the new active load balancer. 3. Broadcast an unsolicited ARP reply from the new active load balancer.

Are you using software like Nginx or haproxy, or a hardware load balancer? If you're using software, then I recommend using something like Heartbeat to share an IP address between the two. It basically does the same thing that mahadri suggested, but it's automatic. Here's how it works.

Let's say you have two servers, primary and secondary -- primary is the one that's currently acting as your load balancer. When both servers are running Heartbeat, they exchange UDP pings every second. If the primary server stops responding, then the Heartbeat process on the backup server will automatically take over the shared IP. When the primary box returns, it won't cause any problems or steal back the IP address, until you explicitly instruct it to do so. (If you really wanted it to steal back the IP automatically, I'm sure you could configure it that way.) You can also initiate a failover with a single command, which is useful when you need to do maintenance on the primary box. Running "/usr/share/heartbeat/hb_takeover all" is a lot simpler than having to manually reassign the IP address.

My company's infrastructure is very dependent on heartbeat -- we use it to support our load balancers and all our back-end services. You need to be careful or it will do "surprising" things when you're rebooting or changing the network config, but on the whole it makes it cheaper and easier to provide highly-available services.

Of course, if you're using a hardware load balancer, it should have a similar failover protocol built-in. I'd recommend just using its built-in features, that's what you're paying the big bucks for.