Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Too much efficiency makes everything worse (2022)

sohl-dickstein.github.io

Post reply on HN
Print view View original

301–310 of 377 posts

Re: Too much efficiency makes everything worse (2022)

#301

Post by remram »

eru wrote: post #28
remram wrote: post #24

Those are great points! Another related law is from queuing theory: waiting time goes to infinity when utilization approaches 100%. You need your processes/machines/engineers to have some slack otherwise some tasks will wait forever.

You can add a measure of robustness to your optimization criteria. You can explicitly optimise for having enough slack in your utilisation to handle these unforeseen circumstances. For example, you can assign priorities to the loads on your systems, so that you can shed lower priority loads to create some slack for emergencies, without having to run your system idle under during lulls. I get what the article is tryin…

A task "shed" is one delivered with infinite latency. If that's fine for you then the theorem doesn't hurt you, do what's best for your domain. It's just something to be aware of.

Re: Too much efficiency makes everything worse (2022)

#302

Post by [deleted] »

vishnugupta wrote: post #69
remram wrote: post #24

Those are great points! Another related law is from queuing theory: waiting time goes to infinity when utilization approaches 100%. You need your processes/machines/engineers to have some slack otherwise some tasks will wait forever.

I feel that a 100% efficient system is not resilient. Even minor disruptions in subsystems lead to major breakdowns. There’s no room to absorb shocks. We saw a drastic version of this during COVID-19 induced supply chain collapse. Car manufacturers had built near 100% just in time manufacturing that they couldn’t absorb chip shortages and it took them years to get back up. It also leaves no room for experimentation.…

[deleted]

Re: Too much efficiency makes everything worse (2022)

#303

Post by EasyMark »

whizzter wrote: post #265

This has become a societal problem in Sweden during the past 20 or so years. 1: Healthcare efficiency is measured by "completed tasks" by primary care doctors, the apparatus is optimized for them handling simple cases and they thus often do some superficial checking and either send one home with some statistically correct medicine (aspirin/antibiotics) or punt away cases to a specialized doctor if it appears to be so…

I wish these were the biggest problems facing US train and healthcare industries.

Re: Too much efficiency makes everything worse (2022)

#304

Post by EasyMark »

t_mann wrote: post #86

The argument rides on the well-known Goodhart's law ( when a measure becomes a target, it ceases to be a good measure ). However, it only puts it down to measurement problems, as in, we can't measure the things we really care about, so we optimize some proxies. That, in my view, is a far too reductionist view of the problem. The problem isn't just about measurement, it's about human behavior. Unlike particles, humans…

I think a big portion of that is humans don’t like to be viewed only as numbers and will rebel and manipulate any system you try to put the thumbscrews to them with. So the quote to mean rings golden and isn’t fallible to much of an extent

Re: Too much efficiency makes everything worse (2022)

#305

Post by thomassmith65 »

crazygringo wrote: post #294
thomassmith65 wrote: post #286

I noticed an example of this rule at my local hardware superstore. Around a decade ago, the store installed anti-theft cages. At first they only kept high-dollar items in the cages. It was a bit of an inconvenience, but not so bad. If a customer is dropping $200+ on some fancy power tool, he or she likely doesn't mind waiting five minutes. But a few years later, there was a change - almost certainly a 'data-driven' c…

> they lock up the items that cumulatively lose the hardware store the most money, due to theft. > I doubt the optimization helped their bottom-line These seem to be in direct contradiction, unless you really think people have stopped going there because of it, to such an extent it outweighs the thefts . Especially when, if they stop going there, the competing local hardware superstore is probably doing the exact sam…

I doubt the optimization helped their bottom-line; I do not know it.

It is possible for a business to make money without customers actually liking the company: hey, it works for some of the FA*NG companies!

That said, there is something that feels 'off' about management obsessing over shrinkage to the point that the shopping experience begins to suck. It's not a truckstop or a drug store in a bad area... it's a hardware superstore.

With too much data, some manager can fixate on $3 screwdriver thievery and not think about the bigger picture: like shoppers finding the store to be a pain in the ass, and therefore no longer an attractive place to buy expensive riding-lawnmowers and floor jacks.

A store can quantify lower sales figures, but it may not be obvious that the lower sales were related to the choice of 'caged vs uncaged' inventory.

But again, I do not know. I only suspect.

Re: Too much efficiency makes everything worse (2022)

#306

Post by AdrianB1 »

nradov wrote: post #215
fireflash38 wrote: post #175

Earlier quoted context omitted.

One of the primary reasons people bail out companies are the knock-on effects. People losing jobs, etc. If society itself is robust enough to cover for people in those situations, we could let companies fail far more.

True, but for some companies there are also national security concerns. If we lose the domestic supply chain for certain items then that limits our freedom of action and leaves us vulnerable to supply disruptions.

If you depend on a single company to supply certain items, you have a big problem already. Pouring money in that company will mostly help the executive bonuses, not the national security.

Re: Too much efficiency makes everything worse (2022)

#307

Post by AdrianB1 »

RandomLensman wrote: post #209
vlovich123 wrote: post #170

Earlier quoted context omitted.

I don’t think it’s smart to proactively back track without being very careful. One thing that’s needed is for corporate death to be allowed to occur. Right now the downsides of risky behavior is bailed out for large enough risk. Then the companies that fail aren’t robust and the ones that don’t are but bailouts let non robust companies keep going. Otherwise “robustness” is a property without a measure which means tha…

Socialized healthcare seems to kind of work in many developed economies - where does it struggle and by what metrics regarding the health outcomes?

I would love to see a good proof that it works, all the discussions, rumors and anecdotal evidence suggest the contrary. I am open to learn the truth, with hard numbers.

Very long waiting times are the first thing that comes to mind regarding such failures, with UK and Canada at the top spot. It is not uncommon to die waiting for a consultation to be diagnosed in 1-2 years.

Re: Too much efficiency makes everything worse (2022)

#308

Post by efitz »

I am skeptical of the analogy to overfitting, although I understand where the author is coming from and agree with the sentiment.

The basic problem is stupid simple. Optimizing a process for one specific output necessarily un-optimizes for everything else.

Right now much of commerce and labor in the United States is over-optimized for humans because tech businesses are optimizing for specific outcomes (productivity, revenue, etc) in a way that ignores the negative impacts on the humans involved.

The optimizations always turn into human goals, eg my manager needs to optimize for productivity if they want a bonus (or not get optimized out themselves), which means they need to measure or estimate or judge or guess each of their employees’ productivity, and stupid MBA shit like Jack Welch’s “fire the lowest 10% every year”) results in horrible human outcomes.

Sure there are people who need to be fired, but making it an optimization exercise enshittified it.

Same for customer service. Amazon wants to optimize revenue. Customer service and returns are expensive. Return too many things? You’re fired as a customer.

Call your mobile providers customer service too often? Fired.

Plus let’s not staff customer service with people empowered to do, well, service. Let’s let IVRs and hold times keep the volumes low.

All anecdotes but you’ve experienced something similar often enough to know it is the rule, not the exception, and it’s all due to over-optimization.

Re: Too much efficiency makes everything worse (2022)

#309

Post by amelius »

lubujackson wrote: post #80
refibrillator wrote: post #55

I recognize the author Jascha as an incredibly brilliant ML researcher, formerly at Google Brain and now at Anthropic. Among his notable accomplishments, he and coauthors mathematically characterized the propagation of signals through deep neural networks via techniques from physics and statistics (mean field and free probability theory). Leading to arguably some of the most profound yet under-appreciated theoretical…

The exciting thing about this idea is if you can correlate, say, economics with the works of ML, that means a computer program which you can run, revise and alter can directly give you measurable data about these complex system interactions that mostly have existed as a platonic idea since reality is too nuanced and multiple to validate concepts formally. With the idea that there is some subset of logic that sits bel…

Well if you can turn chatgpt into an intelligent actor in a simulated economy, and are able to run it at scale, I bet you can get some valuable insights.

Re: Too much efficiency makes everything worse (2022)

#310

Post by amelius »

remram wrote: post #24

Those are great points! Another related law is from queuing theory: waiting time goes to infinity when utilization approaches 100%. You need your processes/machines/engineers to have some slack otherwise some tasks will wait forever.

Slack __or__ lower priority tasks.
Post reply on HN