Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Your Load Balancer Still Sends Traffic to Dead Backends

singh-sanjay.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#11

Post by bastawhiz »

dotwaffle wrote: post #7

I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.

Whether the load balancer connects to the server or reverse, nothing changes. A modern H2 connection is pretty much just that: one persistent connection between the load balancer and server, who initiates it doesn't change much.

The connection being active doesn't tell you that the server is healthy (it could hang, for instance, and you wouldn't know until the connection times out or a health check fails). Either way, you still have to send health checks, and either way you can't know between health checks that the server hasn't failed. Ultimately this has to work for every failure mode where the server can't respond to requests, and in any given state, you don't know what capabilities the server has.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#12

Post by jeffbee »

singhsanjay12 wrote: post #2

I wrote this after seeing cases where instances were technically “up” but clearly not serving traffic correctly. The article explores how client-side and server-side load balancing differ in failure detection speed, consistency, and operational complexity. I’d love input from people who’ve operated service meshes, Envoy/HAProxy setups, or large distributed fleets — particularly around edge cases and scaling tradeoffs…

I don't think you really need sub-millisecond detection to get sub-millisecond service latency. You mainly need to send backup requests, where appropriate, to backup channels, when the main request didn't respond promptly, and your program needs to be ready for the high probability that the original request wins this race anyway. It's more than fine that Client A and Client B have differing opinions about the health of the channel to Server C at a given time, because there really isn't any such thing as the atomic health of Server C anyway. The health of the channel consists of the client, the server, and the network, and the health of AC may or may not impact the channel BC. It's risky to let clients advertise their opinions about backend health to other clients, because that leads to the event where a bad client shoots down a server, or many servers, for every client.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#13

Post by bastawhiz »

owenthejumper wrote: post #10
singhsanjay12 wrote: post #2

I wrote this after seeing cases where instances were technically “up” but clearly not serving traffic correctly. The article explores how client-side and server-side load balancing differ in failure detection speed, consistency, and operational complexity. I’d love input from people who’ve operated service meshes, Envoy/HAProxy setups, or large distributed fleets — particularly around edge cases and scaling tradeoffs…

Modern LBs, like HAProxy, support both active & passive health checks (and others, like agent checks where the app itself can adjust the load balancing behavior). This means that your "client scenario" covering passive checks can be done server side too. Also, in HAProxy (that's the one I know), server side health checks can be in millisecond intervals. I can't remember the minimum, I think it's 100ms, so theoretical…

> theoretically you could fail a server within 200-300ms, instead of 15seconds in your post.

You need to be careful here, though, because the server might just be a little sluggish. If it's doing something like garbage collection, your responses might take a couple hundred milliseconds temporarily. A blip of latency could take your server out of rotation. That increases load on your other servers and could cause a cascading failure.

If you don't need sub-second reactions to failures, don't worry too much about it.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#14

Post by bdangubic »

jayd16 wrote: post #4
AuthAuth wrote: post #3

It seems like passive is the best option here but can someone explain why one real request must fail? So the load balancer is monitoring for failed requests. If it receives one can it not forward the initial request again?

Not every request is idempotent and its not known when or why a request has failed. GETs are ok (in theory) but you can't retry a POST without risk of side effects.

I am a contractor and have been fixing shit large part of my career. non-idempotent POSTs are just about always at the top of the list of shit to fix immediately. To this day (30 years in) I do not understand how can someone design a system where POSTs are not idempotent… I mean I know why, the vast majority of people in our industry are just not good at what they do but still…

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#15

Post by snowhale »

dotwaffle wrote: post #7

I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.

[dead]

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#16

Post by SoftTalker »

bdangubic wrote: post #14
jayd16 wrote: post #4

Earlier quoted context omitted.

Not every request is idempotent and its not known when or why a request has failed. GETs are ok (in theory) but you can't retry a POST without risk of side effects.

I am a contractor and have been fixing shit large part of my career. non-idempotent POSTs are just about always at the top of the list of shit to fix immediately. To this day (30 years in) I do not understand how can someone design a system where POSTs are not idempotent… I mean I know why, the vast majority of people in our industry are just not good at what they do but still…

Yep. I worked in corporate back-office IT way before the web era. It was a requirement that every batch job be re-runable idempotently. So if it failed, you'd identify the bad data, excise it, rerun the job, and deal with the bad record in the morning.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#17

Post by singhsanjay12 »

dotwaffle wrote: post #7

I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.

The idea is attractive (especially for draining), but once you try to map arbitrary inbound client connections onto backend-initiated "reverse" pipes, you end up needing standardized semantics for multiplexing, backpressure, failure recovery, identity propagation, and streaming! So, you're no longer just standardizing "reverse HTTP", you’re standardizing a full proxy transport + control plane. In practice, the ecosystem standardized draining/health via readiness + LB control-plane APIs and (for HTTP/2/3) graceful shutdown signals, which solves the draining problem without flipping the fundamental accept/connect roles.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#18

Post by singhsanjay12 »

firefoxd wrote: post #8
singhsanjay12 wrote: post #2

I wrote this after seeing cases where instances were technically “up” but clearly not serving traffic correctly. The article explores how client-side and server-side load balancing differ in failure detection speed, consistency, and operational complexity. I’d love input from people who’ve operated service meshes, Envoy/HAProxy setups, or large distributed fleets — particularly around edge cases and scaling tradeoffs…

Hi author, a tangent: For us who need to zoom in on mobile devices.

Ok, do you mind briefly describing, what issues you saw on mobile?

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#20

Post by OptionOfT »

jayd16 wrote: post #4
AuthAuth wrote: post #3

It seems like passive is the best option here but can someone explain why one real request must fail? So the load balancer is monitoring for failed requests. If it receives one can it not forward the initial request again?

Not every request is idempotent and its not known when or why a request has failed. GETs are ok (in theory) but you can't retry a POST without risk of side effects.

There were some issues with replaying certain GETs back in the day:

https://news.ycombinator.com/item?id=16964907

Post reply on HN