Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Too much efficiency makes everything worse (2022)

sohl-dickstein.github.io

Post reply on HN
Print view View original

211–220 of 377 posts

Re: Too much efficiency makes everything worse (2022)

#211

Post by immibis »

thomasahle wrote: post #106
refibrillator wrote: post #55

I recognize the author Jascha as an incredibly brilliant ML researcher, formerly at Google Brain and now at Anthropic. Among his notable accomplishments, he and coauthors mathematically characterized the propagation of signals through deep neural networks via techniques from physics and statistics (mean field and free probability theory). Leading to arguably some of the most profound yet under-appreciated theoretical…

I love the idea of ReZero, basically using a trainable parameter, alpha, in residual layers like this: Deep Network | xi+1 = F(xi) Residual Network | xi+1 = xi + F(xi) Deep Network + Norm | xi+1 = Norm(F(xi)) Residual Network + Pre-Norm | xi+1 = xi + F(Norm(xi)) Residual Network + Post-Norm | xi+1 = Norm(xi + F(xi)) ReZero | xi+1 = xi + αi F(xi) However, I haven't actually seen this used in practice. The papers we ha…

Isn't this already part of F?

Re: Too much efficiency makes everything worse (2022)

#212

Post by immibis »

scott_w wrote: post #84
georgeburdell wrote: post #35

Earlier quoted context omitted.

Yep, I used to work in a factory. Target utilization at planning time was 80%. If you over-predict your utilization, you waste money. If you under-predict, a giant queue of “not important” stuff starts to develop

This reminds me of something my mother told me she aimed for when she ran her catering businesses: she always wanted 1 serving of pie leftover at the end of every day. If she had 0, she ran the risk of turning customers away and losing money. Any more than 1 is excess waste. Having just 1 meant she’d served every possible customer and only “wasted” 1 slice.

Customers don't want to buy the last one.

Re: Too much efficiency makes everything worse (2022)

#213

Post by CuriouslyC »

wongarsu wrote: post #192
vlovich123 wrote: post #170

Earlier quoted context omitted.

I don’t think it’s smart to proactively back track without being very careful. One thing that’s needed is for corporate death to be allowed to occur. Right now the downsides of risky behavior is bailed out for large enough risk. Then the companies that fail aren’t robust and the ones that don’t are but bailouts let non robust companies keep going. Otherwise “robustness” is a property without a measure which means tha…

In a free market economy we shouldn't demand robustness, we should create a system that promotes and rewards robustness. A strict commitment against bail-outs would certainly be part of that. Companies (and private people) can decide to lower their risk exposure (at the cost of efficiency/profit) or take out insurance against risks. And if they go the insurance route they have to assess how likely their insurance is…

That solution is never going to work when black swan events occur on the order of every 5-10 years and executive vision is focused on the next quarter with little concern paid to anything outside the next 2-3 years. Nobody is going to want to give up short term performance to mitigate risks that probably won't manifest until after they've left for a better job.

Re: Too much efficiency makes everything worse (2022)

#214

Post by netcan »

bob1029 wrote: post #187
netcan wrote: post #176

Another example of this approximate law is in exercise physiology. To a normal person, there are a lot of good proxy indicators of fitness. You could train sprinting. You could hop up and down. Squat. Clean and jerk.. etc. Running faster,hopping higher, squatting heavier... all indicators of increasing fitness... and success of your fitness training. Two points: 1 - The more general your training methodology, the mor…

Clean & jerk is one of those movements that I would almost consider "complete". Especially if you are going to mix in variants of the squat. Not sure these are the best example. I don't know of anyone who can C&J more than their body weight for multiple repetitions who isn't also an absolute terminator at most other meaningful aspects of human fitness. Human body is one machine. Hormonal responses are global. Enduran…

Perhaps. And you could probably test this but I would gamble that the principle still applies. IE, these weightlifters are probably also very capable (eg) shotputters because of all that weightlifting. But also... their shotput, sprinting and other tangential abilities probably peak at some point. From then on, they are mostly just getting stronger at clean and jerk.

> Hormonal responses are global. Endurance/strength is a spectrum but the whole body goes along for the ride.

This is true, and that is why most exercise is a general good for most people, and has similar physiological effects. However, at some point "specialization" (term of art), kicks in. At that point, a bigger clean and jerk no longer equates to a longer shot put.

Fwiw... This isn't a point about exercise or how to exercise. Most people aren't that specialized or advanced in a sport and the ones who are have coaches. My point is that the phenomenon speculated to be broad in this post applies (I suspect) to physiology. Probably quite broadly. It's just easy to think about it in terms of sports because "training & optimization" directly apply.

Re: Too much efficiency makes everything worse (2022)

#215

Post by nradov »

fireflash38 wrote: post #175
vlovich123 wrote: post #170

Earlier quoted context omitted.

I don’t think it’s smart to proactively back track without being very careful. One thing that’s needed is for corporate death to be allowed to occur. Right now the downsides of risky behavior is bailed out for large enough risk. Then the companies that fail aren’t robust and the ones that don’t are but bailouts let non robust companies keep going. Otherwise “robustness” is a property without a measure which means tha…

One of the primary reasons people bail out companies are the knock-on effects. People losing jobs, etc. If society itself is robust enough to cover for people in those situations, we could let companies fail far more.

True, but for some companies there are also national security concerns. If we lose the domestic supply chain for certain items then that limits our freedom of action and leaves us vulnerable to supply disruptions.

Re: Too much efficiency makes everything worse (2022)

#216

Post by wongarsu »

CuriouslyC wrote: post #213
wongarsu wrote: post #192

Earlier quoted context omitted.

In a free market economy we shouldn't demand robustness, we should create a system that promotes and rewards robustness. A strict commitment against bail-outs would certainly be part of that. Companies (and private people) can decide to lower their risk exposure (at the cost of efficiency/profit) or take out insurance against risks. And if they go the insurance route they have to assess how likely their insurance is…

That solution is never going to work when black swan events occur on the order of every 5-10 years and executive vision is focused on the next quarter with little concern paid to anything outside the next 2-3 years. Nobody is going to want to give up short term performance to mitigate risks that probably won't manifest until after they've left for a better job.

5-10 years is a perfectly normal investment horizon, and in the end investors are the ones electing the CEO and setting goals and rewards for the executive. If betting on the long term is a winning strategy companies absolutely have the means to do that. But right now it usually isn't.

Re: Too much efficiency makes everything worse (2022)

#217

Post by nradov »

bumby wrote: post #195
tablatom wrote: post #98

Earlier quoted context omitted.

Interesting timing for me! Just a couple of days ago I discovered the work of biologist Olivier Hamant who has been raising exactly this issue. His main thesis is that very high performance (which he defines as efficacy towards a known goal plus efficiency) and very high robustness (the ability to withstand large fluctuations in the system) are physically incompatible. Examples abound in nature. Contrary to common pe…

There are also a lot of engineering examples where the goal is to optimize for reliability. I think the most common domain is marine platforms where it is prohibitively expensive to induct and repair (you have to send a team out by helicopter, for example).

And yet most large merchant ships are designed with a single engine, propeller, and rudder to optimize for cost instead of reliability. We have seen some spectacular failures of that approach recently, although it probably still makes sense in aggregate.

A major mechanical casualty beyond what the crew can repair usually means a tow to a shipyard. Flying more engineers in by helicopter would seldom help, and often isn't feasible.

Re: Too much efficiency makes everything worse (2022)

#218

Post by eru »

082349872349872 wrote: post #145
eru wrote: post #62

Earlier quoted context omitted.

For some scenarios that's fine, and you can slash the queue whenever necessary. Eg at Google (this was ten years ago or so), we could always spend leftover networking capacity on syncing a tiny bit faster and more often between our data centres. And that would improve users' experience slightly, but it also not something that builds up a backlog. At a factory, you could always have some idle workers swipe the floor a…

Unlike merchantmen, naval vessels were crewed at a level allowing for substantial attrition (bad attrition would be casualties; good attrition would be prize crews); I believe they traditionally (pace Churchill) had many, many activities which were incidental to force projection (eg polishing the brightwork) but could be used to occupy all hands.

Yes. And, well, you can also always train more. Especially in the age of sail.

Re: Too much efficiency makes everything worse (2022)

#219

Post by eru »

paganel wrote: post #105
eru wrote: post #26

Just add some measure of robustness to your optimization criterion. That includes having some slack for unforeseen circumstances.

And then you optimize around the slack, and we're back to step 1.

The slack is part of your optimisation criteria.

Re: Too much efficiency makes everything worse (2022)

#220

Post by eru »

sahmeepee wrote: post #123
eru wrote: post #26

Just add some measure of robustness to your optimization criterion. That includes having some slack for unforeseen circumstances.

I thought that was the purpose of adding noise (in the mitigations).

Noise is one possible way, but not the only one.
Post reply on HN