This is more a meta comment about the blog itself (as is customary for HN): I like the blog, there has been a lot of work put into it, so it makes me sad that it's hosted on GitHub pages using a subdomain of GitHub.io. When the day comes that GitHub inevitably kills/ruins Pages, because it _will_ happen, there is no question, the links to this blog will be stuck forever pointing to this dead subdomain that the author has no control over. We just have to hope that the replacement blog is findable via search engines, and hope that comments are enabled wherever the pages link is referenced so that new people can find the blog. An unfortunate mess that is definitely going to happen, entirely Microsoft's fault.
Too much efficiency makes everything worse (2022)
101–110 of 377 posts
Re: Too much efficiency makes everything worse (2022)
#102This is more a meta comment about the blog itself (as is customary for HN): I like the blog, there has been a lot of work put into it, so it makes me sad that it's hosted on GitHub pages using a subdomain of GitHub.io. When the day comes that GitHub inevitably kills/ruins Pages, because it _will_ happen, there is no question, the links to this blog will be stuck forever pointing to this dead subdomain that the author…
Strengthening the importance of the Archive (the Wayback).
Re: Too much efficiency makes everything worse (2022)
#103Important subject, so-so blog post. This idea deserves further development. The author seems to be discussing optimizing for the wrong metric. That's not a problem of too much efficiency. Excessive efficiency problems are different. They come from optimizing real output at the expense of robustness. Just-in-time systems have that flaw. Price/performance is great until there's some disruption, then it's terrible for a…
Very good characterisation of close, but distinct concepts. (a map of a domain)
If we squint a little, focus on close/far-away instead of same/distinct and s/metric/model/g (because usage of a metric implies a model), we can see how close these things can be.
Optimizing for the wrong metric - becomes “using a wrong model”.
Excessive efficiency - is partially “using a wrong model”, or maybe “good model != perfect model”. We start with good enough model, but after certain threshold we get to experience the difference between “good enough” and “perfect” (aparantly we care about redundancy, but it was not part of our model; so we were using a wrong model)
Overfitting is “finding the wrong model” (I wanted a model for the whole population, got a model only for a sample)
..or if we squint even more and go meta.. overfitting is part of “good model != perfect (meta)model” of modeling. (using sample data is good enough, but not perfect)
P.S. I liked the article. Choice of the title - not so much.
P.P.S. Simplicity of a model is part of meta-model.
Re: Too much efficiency makes everything worse (2022)
#104The argument rides on the well-known Goodhart's law ( when a measure becomes a target, it ceases to be a good measure ). However, it only puts it down to measurement problems, as in, we can't measure the things we really care about, so we optimize some proxies. That, in my view, is a far too reductionist view of the problem. The problem isn't just about measurement, it's about human behavior. Unlike particles, humans…
This is true, these "laws" are approximations and imperfect reductions.
Which one is useful or descriptive will depend on the specific example.
Optimizing ML VS Optimizing a social media algorithm VS using standardized testing to optimize education systems.
There is no perfect abstraction that applies to these different scenarios precisely. We don't need that precision. We just need the subsequent intuition about where these things will go wrong.
Re: Too much efficiency makes everything worse (2022)
#105Just add some measure of robustness to your optimization criterion. That includes having some slack for unforeseen circumstances.
And then you optimize around the slack, and we're back to step 1.
Re: Too much efficiency makes everything worse (2022)
#106I recognize the author Jascha as an incredibly brilliant ML researcher, formerly at Google Brain and now at Anthropic. Among his notable accomplishments, he and coauthors mathematically characterized the propagation of signals through deep neural networks via techniques from physics and statistics (mean field and free probability theory). Leading to arguably some of the most profound yet under-appreciated theoretical…
I love the idea of ReZero, basically using a trainable parameter, alpha, in residual layers like this:
Deep Network | xi+1 = F(xi)
Residual Network | xi+1 = xi + F(xi)
Deep Network + Norm | xi+1 = Norm(F(xi))
Residual Network + Pre-Norm | xi+1 = xi + F(Norm(xi))
Residual Network + Post-Norm | xi+1 = Norm(xi + F(xi))
ReZero | xi+1 = xi + αi F(xi)
Am I missing something?
Re: Too much efficiency makes everything worse (2022)
#107Earlier quoted context omitted.
Yep, I used to work in a factory. Target utilization at planning time was 80%. If you over-predict your utilization, you waste money. If you under-predict, a giant queue of “not important” stuff starts to develop
This reminds me of something my mother told me she aimed for when she ran her catering businesses: she always wanted 1 serving of pie leftover at the end of every day. If she had 0, she ran the risk of turning customers away and losing money. Any more than 1 is excess waste. Having just 1 meant she’d served every possible customer and only “wasted” 1 slice.
And then you can eat the pie as a reward.
Re: Too much efficiency makes everything worse (2022)
#108[deleted]
Re: Too much efficiency makes everything worse (2022)
#109I was trying to remember where I remember where I heard of this author's name before. Invented the first generative diffusion model in 2015. https://arxiv.org/abs/1503.03585
And for me it was this ingenious 2019 paper co-authored by Stephan Hoyer and Sam Greydanus on doing structural optimization by employing a (constrained) neural network as a storage/modifier/tuner of the physical model describing the structure to optimize: https://arxiv.org/abs/1909.04240
Super interesting approach and very well written paper.
Re: Too much efficiency makes everything worse (2022)
#110Maybe I'm misunderstanding this but this doesn't seem like an accurate explanation of overfitting:
> In machine learning (ML), overfitting is a pervasive phenomenon. We want to train an ML model to achieve some goal. We can't directly fit the model to the goal, so we instead train the model using some proxy which is similar to the goal
One of the pernicious aspects of overfitting is it occurs even if you can perfectly represent your goal via a training metric. In fact it's even worse simetimes as an incorrect training metric can indirectly help regularise the outcome.