When robustly tolerable beats precariously optimal (2020)
1–10 of 24 posts
Re: When robustly tolerable beats precariously optimal (2020)
#2Great short read on a concept we can find in many places, including software engineering or even UX design in general.
Re: When robustly tolerable beats precariously optimal (2020)
#3It's an argument against such things as HTTP/3. That yields a slight increase in performance (maybe), for which there's a large increase in complexity.
Classic issue in military and industrial equipment, where you often accept somewhat less than maximum possible performance in exchange for robustness.
Mechanical designers think about this a lot, because their enemies are wear, vibration, and fragility.
Re: When robustly tolerable beats precariously optimal (2020)
#4This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethality of competition. Figuring out systems that work for your level of risk tolerance is very achievable and can be incrementally improved, but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable
Re: When robustly tolerable beats precariously optimal (2020)
#5I learned this in a circuits class I took back in college, around 1984. Specifically, it was about amplifier circuits an op-amp design. Such designs are a puzzle of tradeoffs, and the teacher emphasized that often the "optimal" design is an inferior design in the light of real world constraints.
The globally optimal point on whatever thing we were optimizing might indeed be the highest peak of the graph, but if it is a sharp peak, any deviation in the voltage, temperature, or the real world values of the components would put the operating point far down the slope of that peak.
It was much better to find a reasonable operating point that had low sensitivity to voltage/temperature/component values but had acceptable behavior (gain, noise, whatever was important).
The surprising thing I learned from that class is that even though resistor and capacitor values and the gain of individual transistors of IC op-amps is an order of magnitude worse than for discrete designs, the matching of those terrible components was an order of magnitude better than for discrete components. Designers came up with many clever ways to take advantage of that to wring terrific performance from terrible components.
For example, say the nominal value of a given resistor in the design might be 4K ohms, and in the discrete design they might be 2%, or 1%, or 0.5% off (the ones with tighter tolerance get ever more expensive), while in the monolithic design the tolerance might be +/- 20%. But all the resistors would be off by the same amount and would match each other to a fraction of a percent, even across temperature and voltage variations.
The other funny effect is that when you buy a discrete 2% tolerance resistor, the distribution isn't gaussian around the mean. That is because the manufacturers have measured all of them and the ones within 0.5% get marked up and put in the 0.5% bin, and the remaining ones within 1% tolerance get marked up less and get put in the 1% bin. As a result, the distribution is bimodal on either side of the "hole" in the middle.
Re: When robustly tolerable beats precariously optimal (2020)
#6It all depends on your the definition of the 'loss' function. One can actually include robustness/sensitivity as the goal and optimize for that.
Re: When robustly tolerable beats precariously optimal (2020)
#7It all depends on your the definition of the 'loss' function. One can actually include robustness/sensitivity as the goal and optimize for that.
And run minimax on a cloud of perturbations surrounding each world model, and then weight them together in a tree after that.
Re: When robustly tolerable beats precariously optimal (2020)
#8Don't let the perfect be the enemy of the good.
Re: When robustly tolerable beats precariously optimal (2020)
#9I learned this in a circuits class I took back in college, around 1984. Specifically, it was about amplifier circuits an op-amp design. Such designs are a puzzle of tradeoffs, and the teacher emphasized that often the "optimal" design is an inferior design in the light of real world constraints. The globally optimal point on whatever thing we were optimizing might indeed be the highest peak of the graph, but if it is…
> measured all of them and the ones within 0.5% get marked up and put
in the 0.5% bin, and the remaining ones within 1% tolerance get
marked up less and get put in the 1% bin. As a result, the
distribution is bimodal on either side of the "hole" in the middle.
That's a great explanation of how iterative tolerance filtering works and why that happens!
Re: When robustly tolerable beats precariously optimal (2020)
#10This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethal…
> especially when existential risk is on the table for entities who
lose a competition.
This really just explains externalities. Competition creates apparent short-range existential risk to a business. Real existential risk (to people or things damaged by your defective product because of "precarious perfection") land somewhere else - usually where their impact is much larger.