21–30 of 44 posts
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
This sounds a lot like the control-theory problem of balancing the proportional, integral, and derivative coefficients of a PID controller [1]? I'm curious how you reached this condition as a requirement: > The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes! It makes sense intuitively , but I'm having trouble proving to myself t…
Picking a metric that reacts to changes quickly is neither necessary nor sufficient, but it certainly helps reduce the error on your calculation. You need to know how far off your set point you are and so you need as accurate a measurement as possible.
Imagine a cruise control for a car where the speedometer had a five second delay. You’d still stay at your desired speed on average but it would vary a lot more and require more work to get back to the desired speed. It would have to accelerate harder and brake harder.
Sorta related: I managed a team that built a 5x 1000 node distributed setup 10+ years ago. We ended up going with a) short DNS TTL + a custom DNS server that sent people to the closest cluster (with some intra-communication to avoid sending people to broken clusters) b) in each cluster; three layers: 1) Linux keepalived load balancing, 2) Our custom HTTP/TLS-level loadbalancers (~20 nodes per DC), 3) our application…
Sorta related: I managed a team that built a 5x 1000 node distributed setup 10+ years ago. We ended up going with a) short DNS TTL + a custom DNS server that sent people to the closest cluster (with some intra-communication to avoid sending people to broken clusters) b) in each cluster; three layers: 1) Linux keepalived load balancing, 2) Our custom HTTP/TLS-level loadbalancers (~20 nodes per DC), 3) our application…
What in god's good name were you guys hosting that required 5,000 quad-socket physical hosts!?
Basically one CPU second per web page. 150k pages/second @ peak. 5 million HTTP requests/s. 150 Gbit/s. The web for 250 million people.
Kinda insane numbers when I think about it now, still. (I left five years ago, after it peaked.)
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
The claim is that it results in pretty decent behavior under load while avoiding some problems caused by delayed information.
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
Adding hysteresis definitely helps to stabilize issues like this. Using rolling windows or exponentially decayed weighting has worked out well in my experience. In general, it seems like load based routing can be quite perfidious if you get the heuristic for “load” wrong. I worked on a system that used total connections as our heuristic, measured by the load balancer. The problem we experienced was that some failure…
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
They work better when it's just one in front of a fleet of servers, and so have the total picture of what is going on, but of course that's quite the bottleneck. So you get two LBs, or more, and they each only have their notion of what the back end fleet is doing. There's no standard feedback mechanism to them at all.
Some offer approaches like measuring response time, but that doesn't work so great as soon as you consider APIs where no two requests perform the same. Was it a fast request that got answered slowly (back-end overloaded?), or a slow request that got answered quickly (back-end bored?). Who knows.
For the service I was working on a few years ago, no two requests are the same by any stretch of the imagination, even for the same API call, and came with a variation on request size, and computational power required to process them.
As you'd expect, traditional loadbalancer behaviour actually handled about things to an okay degree probably 90% of the time. That 10% was a real killer though.
Earlier quoted context omitted.
What in god's good name were you guys hosting that required 5,000 quad-socket physical hosts!?
A popular server-assisted mobile browser for crappy phones. Basically one CPU second per web page. 150k pages/second @ peak. 5 million HTTP requests/s. 150 Gbit/s. The web for 250 million people. Kinda insane numbers when I think about it now, still. (I left five years ago, after it peaked.)