Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Your Load Balancer Still Sends Traffic to Dead Backends

singh-sanjay.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#21

Post by igor47 »

Back in the day, I thought about this problem domain a lot! I even wrote and open-sourced a service discovery framework called SmartStack, an early precursor to later approaches like Envoy, described here: https://medium.com/airbnb-engineering/smartstack-service-dis...

This was a client side framework, in the OPs parlance. What's missing in OP is the insight that the server-side load balancer can also fail -- what will load balance the load balancers? We performed registration based on health checks from a sidecar, and then we also did client side checks which we called connectivity checks. Multiple client instances can disagree about the state of the world because network partitions actually can result in different states of the world for different clients.

Finally, you do also still need circuit breakers. Health checks are generally pretty broad, and when a single endpoint in a service begins having high latency, you don't want to bring down the entire client service with all capacity stuck making requests to that one endpoint. This specific example is probably more relevant to the old days of thread and process pools than to modern evented/async frameworks, but the broader point still applies

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#22

Post by singhsanjay12 »

umairnadeem123 wrote: post #19

[dead]

Agree - sliding window error rates plus client-side circuit breakers (with half-open probes and ramp-up) work really well in practice, and the recovery-speed point is especially important.

The only nuance I was trying to call out is what happens at very large scale. These mechanisms operate per client instance, so each client needs a few failures before it trips its breaker and then runs its own probes and ramp-up. That's perfectly reasonable locally, but when you have hundreds or thousands of clients, the aggregate "learning traffic" can still be noticeable. Each client might only send a little bad traffic before reacting, but multiplied across the fleet it can still add up. Similarly, recovery can still produce smaller synchronized ramps as many clients independently notice improvement around the same time.

So I tend to think of client-side circuit breakers as necessary but not always sufficient at scale. They're great for fast local containment and tail-latency protection, but they work best when paired with some shared signal (LB, mesh control plane, or similar) that can dampen the aggregate effect and smooth recovery globally.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#23

Post by singhsanjay12 »

igor47 wrote: post #21

Back in the day, I thought about this problem domain a lot! I even wrote and open-sourced a service discovery framework called SmartStack, an early precursor to later approaches like Envoy, described here: https://medium.com/airbnb-engineering/smartstack-service-dis... This was a client side framework, in the OPs parlance. What's missing in OP is the insight that the server-side load balancer can also fail -- what wi…

> when a single endpoint in a service begins having high latency

Yes, have seen this first hand. Tracking the latency per endpoint in a sliding window helped in some way, but it created other problems for low qps services.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#24

Post by singhsanjay12 »

dastbe wrote: post #9

kind of right, kind of wrong * for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes. * for server-side load balancing, it's entirely possible to shard forw…

For client-side LB, moving active healthcheck outside into dedicated service, wouldn't it create more reliability issues with one more service to worry about? Are there any examples of this approach being used in the industry?

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#26

Post by gbuk2013 »

I have to say I am not a fan of doing this on the client side.

API gateways (which is what server side load-balancer can be abstracted as) serve as important control points for service traffic, for example for auth, monitoring and observability, application firewall, rate limiting etc.

In my general experience code running on the client side is less reliable due to permutations of browsers, flaky networks, challenges with observability.

That said, client side already has one type of load balancing - DNS - but that doesn’t address the availability challenge.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#27

Post by willi59549879 »

singhsanjay12 wrote: post #18
firefoxd wrote: post #8

Earlier quoted context omitted.

Hi author, a tangent: For us who need to zoom in on mobile devices.

Ok, do you mind briefly describing, what issues you saw on mobile?

Zoom on mobile is not possible. So all the graphs are tiny and not readable.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#28

Post by dastbe »

singhsanjay12 wrote: post #24
dastbe wrote: post #9

kind of right, kind of wrong * for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes. * for server-side load balancing, it's entirely possible to shard forw…

For client-side LB, moving active healthcheck outside into dedicated service, wouldn't it create more reliability issues with one more service to worry about? Are there any examples of this approach being used in the industry?

From a dataplane perspective, it does mean your healthchecks are running from a different location than your proxy. So there are risks where routability is impacted for proxy -> dest but not for healthchecker -> dest.

For general reliability, you can create partitions of checkers and use quorum across partitions to determine what the health state is for a given dest. This also enables centralized monitoring to detect systemic issues with bad healthcheck configuration changes (i.e. are healthchecks failing because the service is unhealthy or because of a bad healthchecker?)

In industry, I personnaly know AWS has one or two health-check-as-a-service systems that they are using internally for LBs and DNS. Uber runs its own health-check-as-a-service system which it integrates with its managed proxy fleet as well as p2p discovery. IIRC Meta also has a system like this for at least some things? But maybe I'm misremembering.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#29

Post by donavanm »

singhsanjay12 wrote: post #24
dastbe wrote: post #9

kind of right, kind of wrong * for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes. * for server-side load balancing, it's entirely possible to shard forw…

For client-side LB, moving active healthcheck outside into dedicated service, wouldn't it create more reliability issues with one more service to worry about? Are there any examples of this approach being used in the industry?

IME you end up with both; something like discrete client, LB, and controller. You can’t rely on any one component to “turn itself off.“ ex a client or LB can easily get into a “wedged” state where it’s unable to take itself out of consideration for traffic. For example, I’ve had silly incidents based on bgp routes staying up, memory errors/pressure preventing new health check results from being parsed, the file systems is going read only, SKB pressure interfering with pipes, and of course, the classic difference between a dedicated health check in point versus actual traffic. All those examples it prevents the client or LB from removing itself from the traffic path.

An external controller is able to safely remove traffic from one of the other failed components. In addition the client can still do local traffic analysis, or use in band signaling, to identify anomalous end points and remove itself or them from the traffic path.

Good active probes are actually a pretty meaningful traffic load. It was a HUGE problem for flat virtual network models like a heroku a decade ago. This is exacerbated when you have more clients and more in points.

As a reference, this distributed model it is what AWS moved to 15 years ago. And if you look at any of the high throughput clouds services or CDNs they’ll have a similar model.

Re: Why Your Load Balancer Still Sends Traffic to Dead Backends

#30

Post by dastbe »

donavanm wrote: post #29
singhsanjay12 wrote: post #24

Earlier quoted context omitted.

For client-side LB, moving active healthcheck outside into dedicated service, wouldn't it create more reliability issues with one more service to worry about? Are there any examples of this approach being used in the industry?

IME you end up with both; something like discrete client, LB, and controller. You can’t rely on any one component to “turn itself off.“ ex a client or LB can easily get into a “wedged” state where it’s unable to take itself out of consideration for traffic. For example, I’ve had silly incidents based on bgp routes staying up, memory errors/pressure preventing new health check results from being parsed, the file syste…

one thing to add for passive healthchecking and clientside loadbalancing is that throughput and dilution of signal really matters.

there are obviously plenty of low/sparse call volume services where passive healthchecks would take forever to get signal, or signal is so infrequently collected its meaningless. and even with decent RPS, say 1m RPS distributed between 1000 caller replicas and 1000 callee replicas, that means that any one caller-callee pair is only seeing 1rps. Depending on your noise threshold, a centralized active healthcheck can respond much faster.

There are some ways to improve signal in the latter case using subsetting and aggregating/reporting controllers, but that all comes with added complexity.

Post reply on HN