Live data from Hacker News

When robustly tolerable beats precariously optimal (2020)

askell.blog

11–20 of 24 posts

Re: When robustly tolerable beats precariously optimal (2020)

#11
post #3

It's an argument against such things as HTTP/3. That yields a slight increase in performance (maybe), for which there's a large increase in complexity. Classic issue in military and industrial equipment, where you often accept somewhat less than maximum possible performance in exchange for robustness. Mechanical designers think about this a lot, because their enemies are wear, vibration, and fragility.

See advael's remark above ours on short-termisim. In the commercial digital world, wear vibration and fragility are not the enemy - other companies are.

Until we can move past that silly winner-takes-all incentive we can't have nice things. Most of the genuinely good stuff will be stillborn. We'll always have a 5% vying for perfection in an ever-escaping, unrealistic red queens race, while the bottom 95% suffer a dearth of simply good-enough. How many objectively better search engines than Google died in the ditch of obscurity between 1998 and 2020?

Re: When robustly tolerable beats precariously optimal (2020)

#12

It all depends on your the definition of the 'loss' function. One can actually include robustness/sensitivity as the goal and optimize for that.

Optimizers are very good. Any "smart" change of the loss function will be equally smartly exploited by the optimizer.

Only way to optimize well is to include the uncertainty of your world model into the model.

For travelling salesman, you obviously want to model that certain roads take longer time to travel at different times of day. No tweak of the loss function would allow you to get realistic/robust solutions to TSP.

Re: When robustly tolerable beats precariously optimal (2020)

#13
post #4

This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethal…

> but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable

This is why I remain an engineer and nothing more. If this sentence ever became meaningful I guess I would have been over-promoted.

Re: When robustly tolerable beats precariously optimal (2020)

#14

It all depends on your the definition of the 'loss' function. One can actually include robustness/sensitivity as the goal and optimize for that.

Optimizers are very good. Any "smart" change of the loss function will be equally smartly exploited by the optimizer. Only way to optimize well is to include the uncertainty of your world model into the model. For travelling salesman, you obviously want to model that certain roads take longer time to travel at different times of day. No tweak of the loss function would allow you to get realistic/robust solutions to T…

Why not? Just adding 2SDs to the travel time of each road will get it to try to avoid the roads that get the worst.

Re: When robustly tolerable beats precariously optimal (2020)

#15

I learned this in a circuits class I took back in college, around 1984. Specifically, it was about amplifier circuits an op-amp design. Such designs are a puzzle of tradeoffs, and the teacher emphasized that often the "optimal" design is an inferior design in the light of real world constraints. The globally optimal point on whatever thing we were optimizing might indeed be the highest peak of the graph, but if it is…

This brings flashbacks of standing next to part bins in circuit labs with a multimeter in hand, trying to match components. Unfortunately, the TAs and course instructors did not emphasize your point, and the measured part values and thus the measured filter cutoffs had to be close to perfect to be checked off.

In the past, I have looked at using optimizers to solve for component values of complex analog circuits. I only looked at optimizing accuracy at one corner, but it would be interesting to see what people have done to optimize multiple variables, like including noise, for multiple corners. I think I've seen some Monte Carlo simulation mentioned once to suggest fuzzy solutions within some specification. I would be curious to see if others know more about this.

Re: When robustly tolerable beats precariously optimal (2020)

#16
post #3

It's an argument against such things as HTTP/3. That yields a slight increase in performance (maybe), for which there's a large increase in complexity. Classic issue in military and industrial equipment, where you often accept somewhat less than maximum possible performance in exchange for robustness. Mechanical designers think about this a lot, because their enemies are wear, vibration, and fragility.

TCP must be destroyed. It's totally ridiculous that the default behavior of video players on flaky wifi networks is that they'll open a separate TCP connection for each video segment, and then when one of those connections inevitably decides that the link's bandwidth is like 1kbps, the whole video will stall for a minute or until the player skips the segment.

HTTP/2 gets you this behavior less often, but when you get unlucky with packet loss it affects every segment instead of just one, so the player doesn't have any segments to skip forward to.

Re: When robustly tolerable beats precariously optimal (2020)

#17

I learned this in a circuits class I took back in college, around 1984. Specifically, it was about amplifier circuits an op-amp design. Such designs are a puzzle of tradeoffs, and the teacher emphasized that often the "optimal" design is an inferior design in the light of real world constraints. The globally optimal point on whatever thing we were optimizing might indeed be the highest peak of the graph, but if it is…

Surely anyone doing analogue design for serial or mass production would do a sensitivity analysis, even in 1984. Now it should be much easier to repeatedly randomise the values of every part within their tolerance and run a simulation to check that the end product remains in spec.

You might even do this in order to find out if wider tolerance parts would be good enough and save a little money.

Re: When robustly tolerable beats precariously optimal (2020)

#18
Curiously, this is a big facet in our dev/ops re-organization.

For example, we in infra-operations are responsible to store data customers upload into your systems. This data has to be considered not reproducible, especially if it's older than a few days. If we lose it, we lose it for good and then people are disappointed and turn angry.

As such, large scale data wipes are handled very carefully with manual approvals from several different teams. The full deletion of a customer goes through us, account management, contract and us again. And this is fine. Even with the GDPR and such, it is entirely fine that deleting a customer takes 1-2 weeks. Especially because the process has caught errors in other internal processes, and errors in our customers processes. Suddenly you're the hero vendor if the customer goes "Oh fuck, noooooo".

On the other hand, stateless code updates without persistence changes are supposed to be able to move as fast as the build server gives. If it goes wrong, just deploy a fix with the next build or roll back. And sure you can construct situations in which code changes cause big, persistent, stateful issues, but these tend to be rare with a decent dev-team.

We as infra-ops and central services need to be robust and reliable and are fine shedding speed (outside of standard requests) for this. A dev-team with a good understanding of stateful and stateless changes should totally be able to run into a wall at full speed since they can stand back up just as quickly. We're easily looking at hours of backup restore for hosed databases. And no there is no way to speed it up without hardware changes.

Re: When robustly tolerable beats precariously optimal (2020)

#19
post #4

This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethal…

> but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable This is why I remain an engineer and nothing more. If this sentence ever became meaningful I guess I would have been over-promoted.

lol okay I'll admit upon reading it again that that was a more verbose way to make that point than it needed to be

How about "prioritizing robustness and other kinds of long-termism need to be at least somewhat protected by the system of incentives they operate in to succeed"? I invoked fractal self-similarity there because I think you best protect long-termism by prioritizing long-termism in decision-making at scopes further out

Re: When robustly tolerable beats precariously optimal (2020)

#20

I learned this in a circuits class I took back in college, around 1984. Specifically, it was about amplifier circuits an op-amp design. Such designs are a puzzle of tradeoffs, and the teacher emphasized that often the "optimal" design is an inferior design in the light of real world constraints. The globally optimal point on whatever thing we were optimizing might indeed be the highest peak of the graph, but if it is…

That last part is a bit of a myth, or at least it's something that doesn't consistently happen, at least nowadays. Binning parts is expensive, and for something like a resistor it's rarely worth it, so it's much more likely it's just a well dialed-in manufacturing process and design which stays within that tolerance.
Post reply on HN