Where has all the productivity gone?

Yeah this also sounds like part of the "MBA-ification" of business.

I think a lot of this comes from a reasonable goal to have: if can find appropriate metrics which represent the work your company is doing, this should give you a reasonably objective way of measuring how different interventions move you closer to or farther from your goal.

But I think the problem is that actually finding and implementing those metrics is a bit of a pipe dream in most cases. A lot of these metrics which people come up with to try to measure productivity - story points, ticket flow, time-on-task - are all fairly subjective, and not very good proxies for measuring actual value creation.

I think it gets worse in orgs where there's a disconnect between management and the workforce which is actually getting things done. Often times, upper management only has these artificial metrics to evaluate the performance of a team. A lot of company cultures put management in a bit of a bubble, so middle managers have more of a focus on conversations with other people in that management layer, who are metrics focused, rather than dealing with development teams, who are actually engaged with the product.

As a result, the "ground truth" for management becomes the metrics, and you start to see really silly things happen because the metrics matter more than the product.

Meanwhile, one of the great shocks that I experienced when becoming a manager was that I had no better solution to keeping a large group of people aligned and unblocked than a bunch of meetings. Sure, I tinkered around the edges a bit, but it's not fundamentally different. Solutions welcome!

I have held this view for most of my life. But now I kind of see them as complementary to each other.

At the end of a workday, a spacex or Tesla employee will likely sit down and watch stupid tiktok videos or Netflix or any number of things the rest of us do (or just sleep).

The point is that we need both these things and they do complement each other.

The way I think of it though, is more about scarcity of talent.

The idea that a really smart human being (a finite quantity) can either help build a better renewable energy system or a better video compression pipeline, that feels more obvious. One is using their talents to help humanity. The other is using their talents to help a company’s bottom line but doesn’t actually add to the pleasure of the audience or the enjoyment of humanity like the actual content does. And the cost of having a slightly less efficient video processing pipeline or subtitles software, is negligible compared to the cost of taking another five years to build better toilets, ways to recycle plastic, or use drones to plant trees.

So it’s less about whether the companies are useful. It’s more about where we most need our best and brightest talents used.

It’s that gross misallocation of talent at places like Facebook, Netflix, and Apple that bothers me.

The grandparent talks about going from 1,000 to to 10,000, not a single researcher. Are you implying the law of diminishing returns doesn't apply to research?

I've worked in research most my career and I tend to agree. The issue is that throwing more people in the same narrow approach most likely won't help if that approach is off to begin with. It has been promising and we've been getting and still get some returns out of it, but is it possible we're stuck in a local minima of the solution space in terms of representing the real system were trying to understand and manipulate? We of course can't know and those who are captains of a discipline and the funding agencies that decide who gets funded and don't steer to focus on narrow incremental changes. Some of this has to do with deeper business management steeping into basic scientific research, where I believe, it doesn't belong because it thinks in the wrong time horizons and has the wrong goals.

High risk, novel approaches often aren't funded or few opportunities exist for them. There is good reason for this because it can be abused by those just looking for fun or easy work while labeling it novel. It may also encourage fringe sciences that are bordering on psuedo-scientific work. They incorporate a bit of science but then go off in directions that are almost provably wrong. Distinguishing which novel approaches seem like genuinely novel realistic and non abusive proposals isn't always easy. In some cases they're just laughably wrong because there are so many assumptions baked in that are almost provably wrong or at least self inconsistent. On the other hand, some aren't quite so easy and involve a significant amount of effort and insight to understand exactly what's being proposed. This is often where paradigm shifting research really occurs.

It can take incredibly brilliant scientists with enough creativity to see the opportunity in a proposal and approve it and those aren't the people often awarding proposals. On the flip side, most of these sort of proposals, no matter how valid and novel they may be, simply aren't going to be correct and the reviewer is right to take a more critical eye. The novel approaches are inherently high risk and most will be wrong. I still think we need to fund these approaches. I've been involved in proposals that seemed only slightly novel in direction from accepted paradigms and reviewer responses came back (some agencies return anonymized responses) in a way that showed they clearly didn't understand enough about the domain to even make the assessment when their critiques were clearly invalid, they just didn't agree with the proposal approach. Maybe they were right the overall approach was off and created a lame excuse, maybe they were wrong, but this tendency to be risk averse even in the few funding opportunities that were clearly budgeted to be high risk shows the culture we have in modern science.

The burden of responsibility often lies on those proposing these huge shifts and we may be reaching points in some domains of human knowledge where the burden of proof is simply too high for an individual to provide for a given novel idea. Imagine if Peter Higgs alone had to provide evidence to his idea. It wasn't until so many other iteratative approaches were exhausted that particle physics decided they had to start testing novel other options. How much time, effort, even careers were wasted chasing other ideas? Should science be depth first search, breadth first search, a mixture of the two to try and hedge out bets that if we are on the wrong direction we might find a better paradigm in parallel we can jump to when we're stuck, or perhaps a different heuristic?

Then the really really difficult question, for me: if we support a mixture model of say BFS, DFS (and perhaps a handful of others) for the search space of acquirable knowledge, who should decide resourcing allocations for the search mixture model? What is the best set of approaches and resourcing? Science already incorporates complex search mixture approaches for knowledge at various levels but it seems at the highest level (how we resource stuff) it doesn't, it's incredibly iterative, risk averse, and focused on short time horizons for returns on investment.