Efficiency trades off against resiliency • Hacker News Board

Re: Efficiency trades off against resiliency

#51

Post by bluepizza » Sun, Apr 16, 2023, 3:38 PM UTC

twawaaay wrote: post #39

bluepizza wrote: post #34

Earlier quoted context omitted.

How do you use control theory?

You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes. As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I ca…

This is extremely interesting!

I have been trying to move away from dumb rate limiting to a more holistic approach that allows us to make smarter decisions with traffic. Your overview made me intrigued.

Do you have any references you like to use? I am looking at the Wikipedia page, but it's so removed from practical aspects.

Re: Efficiency trades off against resiliency

#52

Post by LastTrain » Sun, Apr 16, 2023, 3:44 PM UTC

quickthrower2 wrote: post #8

cmilton wrote: post #4

Earlier quoted context omitted.

How is peak load defined in a hospital? (I assume you mean patients. I apologize if that is incorrect.)

For a public health system you run the stats, using the $ cost of a life (or equivalent for a well life vs. sick life). Sounds sinister but ultimately they have so many $ and need to decide how to optimise that money.

You do that for a non-public health system too, but you also figure in margins.

Re: Efficiency trades off against resiliency

#53

Post by angarg12 » Sun, Apr 16, 2023, 3:45 PM UTC

The 100% CPU utilization is an odd one because, although theoretically it's more "efficient" to use a system at full capacity, performance tends to suffer and practically make it unusable (others' pointed here to queueing theory).

We are building a high load, low latency inference system, and we noticed that latency is correlated to CPU usage. Latency grows slowly linearly with CPU usage, up to a point where it starts to grow exponentially. We had a bug where a host was pegged at 100% CPU, and our tail latency suffered greatly, since any request that hit that host was basically toast.

Funnily enough OP could have used examples outside computing where efficiency directly trades off with resiliency. For example, in farming large monocultures are the most efficient, but a single disease can wipe entire plantations. In manufacturing, keeping stock is seen as an inefficiency, but problems in the supply chain can get you in hot water (as COVID made painfully obvious).

Re: Efficiency trades off against resiliency

#54

Post by HALtheWise » Sun, Apr 16, 2023, 3:50 PM UTC

threeseed wrote: post #20

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less ma…

I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in.

Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.

Re: Efficiency trades off against resiliency

#55

Post by HALtheWise » Sun, Apr 16, 2023, 4:01 PM UTC

threeseed wrote: post #21

HALtheWise wrote: post #18

Earlier quoted context omitted.

I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so…

The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.

Good points.

For security concerns, you would probably be selective about which services and workloads you move to cloud compute, possibly preferring to offload internal services like CI before customer data.

Regarding cost, are you implying that there's large monetary cost to preserving the ability to spin up cloud workers in an emergency, or just that once you do it's expensive per day to keep them running? Presumably if you're running an on-prem cluster for cost reasons, you would work to scale it up quickly after an unexpected sustained demand spike, and the cloud supplement would be temporary. If you're getting "unexpected" spikes every week, you need to work harder at forecasting.

In any case, I suspect today that people running on-prem clusters provision them with the intent of being able to handle a load growth and keep weekly average utilization at during the spike. Investing in building infrastructure that keeps working smoothly when weekly average load is 90-100% means you can provision a smaller cluster either way.

Re: Efficiency trades off against resiliency

#56

Post by twawaaay » Sun, Apr 16, 2023, 4:04 PM UTC

bluepizza wrote: post #51

twawaaay wrote: post #39

Earlier quoted context omitted.

You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes. As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I ca…

This is extremely interesting! I have been trying to move away from dumb rate limiting to a more holistic approach that allows us to make smarter decisions with traffic. Your overview made me intrigued. Do you have any references you like to use? I am looking at the Wikipedia page, but it's so removed from practical aspects.

I don't. There simply isn't any tooling or literature to speak of. I have some experience using control engineering in my electronics projects and that's how I came up with the idea to use it for backend systems. I have researched and developed everything myself. I have used "Modern Control Engineering" by Katsuhiko Ogata, but really, mostly I just learned from the Internet.

My initial motivation was to remove configuration. I have found, historically, that giving people options to configure very complex software more often than not results in problems, especially after original developers leave. More often than not these new people will not understand the implications or interactions between various settings and this will just cause problems. So my aim became to remove any options from the software and make sure it can perform autonomously and recover from wide range of, possibly unknown, situations. Which is exactly what control engineering is about if you think about it!

Re: Efficiency trades off against resiliency

#57

Post by theptip » Sun, Apr 16, 2023, 4:09 PM UTC

threeseed wrote: post #21

HALtheWise wrote: post #18

Earlier quoted context omitted.

I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so…

The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.

“Never” is way too strong.

This technique is called “cloud bursting” and is done often enough to have docs on Azure: https://azure.microsoft.com/en-us/resources/cloud-computing-...

More generally, hybrid on/off prem (or multi-cloud) is a top level feature of GCP’s Anthos.

You’re not wrong about your two main reasons, and it’s true that it’s hard to wire up. But you definitely can wire up hybrid workloads in some cases.

The obvious one would be stateless batch compute like image/video processing; “I prefer to re-encode my videos on prem but can burst to the cloud if I get a traffic spike”. This might end up being cheaper than either of all-on-prem with overcapacity, or all off-prem with auto scaling.

I agree it’s not likely to be useful for something where security is your reason for going on-prem.

Re: Efficiency trades off against resiliency

#58

Post by pyth0 » Sun, Apr 16, 2023, 4:11 PM UTC

remram wrote: post #44

threeseed wrote: post #20

Earlier quoted context omitted.

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less ma…

Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask. (scrape interval is 15s, pod creation requires pulling images)

> (scrape interval is 15s, pod creation requires pulling images)

You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.

Re: Efficiency trades off against resiliency

#59

Post by kamma4434 » Sun, Apr 16, 2023, 4:21 PM UTC

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

A long time ago, I was surprised to discover that a mainframe always runs at 100% cpu usage. Its OS makes sure that multiple VMs, each of which has a long list of real-time and batch jobs, neatly prioritized, keep it well fed. An interactive user need an answer now, but a clean-up job can literally take weeks to run. When I looked amazed, the operators told me ‘This thing is so expensive- it cannot be wasted’

Re: Efficiency trades off against resiliency

#60

Post by HALtheWise » Sun, Apr 16, 2023, 4:31 PM UTC

silvestrov wrote: post #42

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

Beware of I/O, memory usage and cache eviction. A low-priority background job can issue I/O requests that result in high I/O latencies for everything else (by e.g. making lot of random seeks). This is very easy to get when using spinning rust.

If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the number of cores allocated to foreground vs background tasks (within less than your latency SLO). That means each core is only doing one kind of work for many milliseconds in a row so CPU caches remain hot, and I/O can be sorted into separate low and high priority queues. IO blocking is much less of a concern with NVMe drvies anyway, since they're intrinsically so parallel.

Memory usage is still a concern, since you do need to have the code and in-memory data for both workloads loaded on some number of your servers in order to rapidly change what tasks they're doing. Physical machines have so much memory today (compared to the code size or working set of most programs) that I suspect this isn't actually a blocker in practice, and it's not necessarily required for every machine in the cluster to have both workloads resident as long as some of them do.

This definitely feels like diving into an alternate universe of cluster management, and I'm not sure how easily Linux gives you tools to do things like this.