Too much efficiency makes everything worse (2022)

Calling this the "strong version of Goodhart's law" was immediately brain-expanding for me.

I have been thinking of goodhart's law a lot, but realized I had been leaning toward focusing on human reaction to the metric as the cause of it; but this reminded me it's actually fundamentally about the fact that any metric itself is inherently not an exact representation of the quality you wish to represent.

And that this may, as OP argues, make goodhart's law fundamental to pretty much any metric used as a target. Independently of how well-intentioned any actors. It's not a result of like human laziness or greed or competing interests, it's an epistemic (?) result of the neccesary imperfection of metric validity.

This makes some of OP's more contentious "Inject noise into the system" and "early stopping" ideas more interesting even for social targets.

"The more our social systems break due to the strong version of Goodhart's law, the less we will be able to take the concerted rational action required to fix them."

Well, that's terrifying.

I do like it when researchers try to connect the deep ideas within their work to broader more general systems, but caution is warranted to the optimism. This article is the kind of formal analogy that inspired/drove much of the marketing appeal of the Santa Fe Institute back in the 1980s. It's honestly always pretty fun, but the devil is usual in the details here (as is usual in making anything "work", such as self-organized criticality [1] which if you enjoyed this article you will also probably enjoy!).

As just one example to make this point more concrete (LOL), the article mentions uncritically that "more complex ecosystems are more stable", but over half a century ago in 1973 Robert May wrote a book called "Stability and Complexity in Model Ecosystems" [2] explaining (very accessibly!) how this is untrue for the easiest ideas of "complex" and "stable". In more human terms, some ideas of "complex" & "stable" can lead you astray, as has been appearing in the relatively nice HN commentary on this article here.

Perhaps less shallowly, things go off the rails fast once you have both multiple metrics (meaning no "objective Objective") and competing & intelligent agents (meaning the system itself has a kind of intrinsic complexity, often swept under the rug by a simplistic thinking that "people are all the same"). I think this whole topic folds itself into "Humanity Complete" (after NP-complete.. a kind of infectious cluster of Wicked Problems [3]) like trust/delegation do [4].

[1] https://en.wikipedia.org/wiki/Self-organized_criticality

[2] https://press.princeton.edu/books/paperback/9780691088617/st...

[3] https://en.wikipedia.org/wiki/Wicked_problem

[4] https://en.wikipedia.org/wiki/Demarcation_problem

I propose to denote the worsening trajectory by the term “enshittification”.

> Unlike particles, humans will actively seek to exploit any control system you've set up.

But that’s only possible because the control system doesn’t exactly (and only) control what we want it to control. The control system is only an imperfect proxy for what we really want, in a very similar way as the measure in Goodhart’s law.

Another variation of that is the law of unintended consequences [0]. There is probably a generalized computational or complex-systems version of it that we haven’t discovered yet.

[0] https://www.sas.upenn.edu/~haroldfs/540/handouts/french/unin...

I recommend the book from Spufford (2010): Red Plenty on this topic.

- https://amp.theguardian.com/books/2010/aug/08/red-plenty-fra...

- https://chris-said.io/2016/05/11/optimizing-things-in-the-us...