Efficiency trades off against resiliency • Hacker News Board

Re: Efficiency trades off against resiliency

#31

Post by [deleted] » Sun, Apr 16, 2023, 9:15 AM UTC

threeseed wrote: post #20

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less ma…

[deleted]

Re: Efficiency trades off against resiliency

#32

Post by pkolaczk » Sun, Apr 16, 2023, 9:26 AM UTC

sjducb wrote: post #26

pclmulqdq wrote: post #17

On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't h…

I would question if the serialisation code speed actually matters. RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk. To speed it up you have to reduce the file size so that it can be written to disk faster. I think the fastest way would be a gzipped JSON file. You can do all of the converting to JSON and compression while you're waiting for the hard disk to fi…

> RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk.

I wouldn't be so sure about it. Current SSDs can do gigabytes per second read and write. Sure, this is order of magnitude slower than theoretical sequential memory throughput, but might not be slower if you need to do some data processing on CPU or if you do memory access in an inefficient way. There are many ways you can screw it up and processing becomes the bottleneck.

For example if you need to compress data, you might be already not be able to saturate SSD (most compression algorithms don't go faster than 1 GB/s maybe except LZ4).

If you're not careful enough with heap allocations and you happen to create lot of tiny objects when deserializing, or you do lot of pointer chasing when serializing you may as well end up very far from theoretical maximum performance. I've seen systems maxing out at 10 MB/s just due to inefficient serde. Using plain text formats makes it somewhat worse.

Re: Efficiency trades off against resiliency

#33

Post by anonymoushn » Sun, Apr 16, 2023, 10:41 AM UTC

sjducb wrote: post #26

pclmulqdq wrote: post #17

On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't h…

I would question if the serialisation code speed actually matters. RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk. To speed it up you have to reduce the file size so that it can be written to disk faster. I think the fastest way would be a gzipped JSON file. You can do all of the converting to JSON and compression while you're waiting for the hard disk to fi…

I think this comment assumes very slow disks compared to the disks that are available.

Re: Efficiency trades off against resiliency

#34

Post by bluepizza » Sun, Apr 16, 2023, 10:43 AM UTC

twawaaay wrote: post #30

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases. The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily. I would like to use this occasion to point out that all…

How do you use control theory?

Re: Efficiency trades off against resiliency

#35

Post by bluepizza » Sun, Apr 16, 2023, 10:46 AM UTC

pclmulqdq wrote: post #17

On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't h…

Serialization/deserialization is still quite costly even with protobufs/flat buffers. This is not the last tiny bit. In some applications, I've seen it take more than 15% of the total CPU usage.

Re: Efficiency trades off against resiliency

#36

Post by 082349872349872 » Sun, Apr 16, 2023, 10:57 AM UTC

twawaaay wrote: post #30

HALtheWise wrote: post #7

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can…

Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases. The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily. I would like to use this occasion to point out that all…

Batching (and sorting and merging) are things our predecessors in the 1950s and 1960s (and before that, in the card era) had to do to run anything at all. These days they are things that we may do to make sluggish systems snappy.

Re: Efficiency trades off against resiliency

#37

Post by steve_gh » Sun, Apr 16, 2023, 11:13 AM UTC

chrsjxn wrote: post #15

100% is an odd choice for a utilization target. Just about every service I've ever monitored has had some fairly clear inflection points where higher utilization starts to affect performance. It could be things like more time spent on garbage collection, or just unlucky collisions with async work that make things take longer.

Basic queueing theory says 100% load is not possible with any sort of variance in the arrival rate or service rate.

Realistically 70% is a good starting point for a max continuous load - then tune it. You might end up a bit lower or higher

Re: Efficiency trades off against resiliency

#38

Post by 1bent » Sun, Apr 16, 2023, 11:14 AM UTC

Tagbert wrote: post #9

bsder wrote: post #3

Maximally efficient is minimally robust.

This also applies to supply chains and economies in the whole.

Indeed, when I saw the headline, I thought this was going to be commenting on the various supply chain crashes triggered by the pandemic's sudden changes, after decades of cleverly squeezing slack out of shipping, ports, warehouses, and logistics coordination.

Re: Efficiency trades off against resiliency

#39

Post by twawaaay » Sun, Apr 16, 2023, 11:24 AM UTC

bluepizza wrote: post #34

twawaaay wrote: post #30

Earlier quoted context omitted.

Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases. The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily. I would like to use this occasion to point out that all…

How do you use control theory?

You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes.

As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I can add a delay to each response and I can have even something as simple as PID controller regulate the CPU usage by changing the dalay. Larger delay will usually cause the clients to slow down requests (requests being usually a result of previous request completing). This is simple and naive example but this is more or less what I do.

(Of course, in reality, it is much better to just have a backpressure mechanism and whenever possible you should use one rather than try to work around HTTP inadequacy. But you can't always do it, especially if you have a public API.)

I also typically have lots of other controllers. For example something that regulates memory usage by limiting transactions in flight or something that regulates latency as seen by priority clients or database replication rate/delay, or error rates or a bunch of other parameters.

I also routinely take care of babysitting downstream systems like databases or other APIs. I may have a regulator that will automatically start backing off certain types of traffic as a response to increasing error rates or latencies in a downstream system. All this because those downstream systems are usually shit and not designed to deal with overload and it is easier for me to deal with this proactively than do what everybody else does -- keep bugging those people to fix their issues when their evidently don't know how.

Re: Efficiency trades off against resiliency

#40

Post by pclmulqdq » Sun, Apr 16, 2023, 11:35 AM UTC

bluepizza wrote: post #35

pclmulqdq wrote: post #17

On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't h…

Serialization/deserialization is still quite costly even with protobufs/flat buffers. This is not the last tiny bit. In some applications, I've seen it take more than 15% of the total CPU usage.

In some applications, this is true. I think 15% is pretty extreme, though, unless what you're doing is something like parsing/ creating data feeds (where the serialization is the point). In those cases, it's probably a good idea to have your own format. Still, JSON would be a lot worse for these cases than protobufs or flatbuffers.

Also, are you sure you aren't compressing them if you're using 15% CPU?