31–40 of 44 posts
Earlier quoted context omitted.
What in god's good name were you guys hosting that required 5,000 quad-socket physical hosts!?
A popular server-assisted mobile browser for crappy phones. Basically one CPU second per web page. 150k pages/second @ peak. 5 million HTTP requests/s. 150 Gbit/s. The web for 250 million people. Kinda insane numbers when I think about it now, still. (I left five years ago, after it peaked.)
Earlier quoted context omitted.
This sounds a lot like the control-theory problem of balancing the proportional, integral, and derivative coefficients of a PID controller [1]? I'm curious how you reached this condition as a requirement: > The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes! It makes sense intuitively , but I'm having trouble proving to myself t…
So I studied control theory after I left Reddit and you’re indeed right, it’s a PID issue. Picking a metric that reacts to changes quickly is neither necessary nor sufficient, but it certainly helps reduce the error on your calculation. You need to know how far off your set point you are and so you need as accurate a measurement as possible. Imagine a cruise control for a car where the speedometer had a five second d…
Thanks for explaining!
Earlier quoted context omitted.
So I studied control theory after I left Reddit and you’re indeed right, it’s a PID issue. Picking a metric that reacts to changes quickly is neither necessary nor sufficient, but it certainly helps reduce the error on your calculation. You need to know how far off your set point you are and so you need as accurate a measurement as possible. Imagine a cruise control for a car where the speedometer had a five second d…
Oh I see! I was misunderstanding what you meant by "quickly"; it seems you're referring to the sensor delay / how tight the feedback loop is. Thanks for explaining!
Delay in the feedback path counts against phase margin, requiring you to reduce the loop bandwidth to maintain stability.
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
> The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes!
The key things are that you don't react to changes in the metric faster than the metric can move (be appropriately damped), and that you react to the metric "smoothly" (e.g. pick a random server where the odds to get a specific server vary with the loading metric, like you mention).
As others say, it's fundamentally a controls problem... a controls problem where there are many, many actuators and the delay/phase shift is relatively unpredictable. Making things easy for the control system by making reaction smooth and the system overall react slowly (overdamped) is important.
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
Per Brendan Gregg's Gesamptkunstwerk , BPF Performance Tools, I feel like you should be able to measure instructions per cycle at the service level. Even in the cloud if exposed by Xen. And even at the resource utilization level for each container. http://www.brendangregg.com/bpf-performance-tools-book.html Of course, you can always just use cloudflare ;)
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
A useful metric for "load" is just as hard as doing the load balancing itself.
Requests can be vastly different, unless you have only one application they're also constantly changing and there are more load balancers involved (both horizontally and vertically as a single request can pass multiple). There are also numerous failure conditions under which responses are very fast.
In that situation it's easier to design for an even spread, then work to improve that metric as much as possible as more information becomes available.
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
I worked for a major cloud service a few years back, and got a wonderful introduction in to how loadbalancers are, for the most part, somewhat awful. They work better when it's just one in front of a fleet of servers, and so have the total picture of what is going on, but of course that's quite the bottleneck. So you get two LBs, or more, and they each only have their notion of what the back end fleet is doing. There…
The intention was that if a server was returning results "quickly" that meant it was least-loaded, and could handle the newest requests.
What it actually meant though was that the server disk filled up, and it started returning "500, Internal Server Error" errors. Very quickly.
At the point the alarms were raised almost all incoming traffic had been routed to this dead/dying host.
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value. Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine w…
We also had to keep track of how many people were on the webpage for a channel when it went live so that we could preemptively replicate the video stream to enough, but not too many, servers.
It seems that in both cases the idea is to distribute (supposedly independent) requests between workers and one of the main difficulties is that requests might not be independent either within one stream (say, in the case of sessions) or between different streams (say, if they need to use one common state).