Elite Underproduction

> You'll also need to pay for the people doing the research to buy houses, have cars, retirement savings, raise families, and have a decent quality of life too.

Newton and Cavendish didn't have any of those, except that Cavendish had retirement savings. So while not having houses, cars, retirement savings, the ability to raise a family, and a decent quality of life might be a reason for not achieving more than Newton or Cavendish, it's not a reason for achieving less.

(I suspect that raising a family is actually counterproductive. I've seen an awful lot of promising researchers of both genders stop publishing after their first baby.)

I don't have any of those things, but while I'm no Newton or Cavendish, I can't attribute the difference to my lack of a car. Most of the people in my country don't have any of those except for houses; most of the people here who own houses built them with their own hands rather than buying them.

It's true that if you have to choose between going into data science and owning your own house and car, or doing quantum optics while living in poverty, the former is a lot better for you. But for most of us it's not an either-or choice:

1. Why not both? Lagrange did a significant amount of his work while subsisting on a day job teaching ballistics to gunners. Vipul Ved Prakash wrote Vipul's Razor by working one month out of the year in Delhi, then spending the other 11 months up in the mountains working on whatever he wanted to. Sidis deliberately didn't do anything others would consider useful, surviving on a series of menial jobs, but if he'd turned his formidable intellect on the problems of automatic computation or chemistry instead of collecting streetcar transfers, possibly he would have made significant progress. I've been living on US$6k a year, so a single year of a US$250k salary and stock package at Google or Fecebutt would allow me to survive for 40 years.

2. For most of us, it's neither. Most people don't have a data-science job available, or cars, or job security, or retirement savings. Given the choice between spending your spare time on watching Westworld and setting up experiments with Kerr cells and third-harmonic-generation crystals, what could possibly make the former a better choice? Though I'm one to talk! Here I am wasting my time commenting on the orange website, arguing about physics with people (who aren't you) whose understanding of physics evidently comes from WGBH Boston.

So, while I agree that everyone should live in material abundance, I don't agree with your apparent conclusion that more abundance of non-experimental-apparatus material goods would boost the world's research productivity. Many more people can buy houses, buy cars, have retirement savings, and raise families than 50 or 100 years ago, while the speed of scientific advancement has increased only modestly.

Yeah, although maybe mentorship can scale reasonably well (or at any rate much faster than we are scaling it at present), I agree with you about collaboration: ten times as many people working on the LHC (or HEP in general) probably wouldn't be very effective. Now that we have Sci-Hub, the General Index, Wikipedia, Stack Exchange, and Google Scholar, we can probably collaborate a little more effectively than before, but not enough to cram orders of magnitude of people into a given subfield.

There might be a path forward in the work on making computational work easily reproducible, by people like Konrad Hinsen, Yihui Xie, Jeremiah Orians, Eelco Dolstra, Ludovic Courtès, Shriram Krishnamurthi, Ricardo Wurmus, and Sam Tobin-Hochstadt, but clearly it hasn't been a panacea so far. Speedrunning results are in many cases reproducible by virtue of nailed-down console hardware and bit-identical game images, but that's harder to achieve even for FEM simulations of turbulent MHD systems, much less actual experimental MHD systems like a Farnsworth fusor.

How do people initially get up to speed on speedrunning? Are there tutorials, the equivalent of a textbook with problem sets, some other onramp? Can we gamify learning quantum mechanics? (I've tried QiskitBlocks but so far haven't been impressed.)

In https://news.ycombinator.com/item?id=29144119 I listed a lot of the big unsolved problems in physics, though many would argue that the question of how consciousness arises isn't part of physics. In https://news.ycombinator.com/item?id=29153894 I also listed a lot of recent advances, many of which came from experimental physics. Some of those problems are susceptible to experimental investigation even with the theories we have.

But, aside from these big problems, many smaller problems are susceptible to experimental investigation. You may not create an Einstein-style paradigm shift by detecting CNO-fusion neutrinos from the sun, observing Higgs decay, improving quantum-dot solar cells, fabricating nanotube rope whose strength approaches that of a single nanotube, understanding the lubricity of BAM well enough to design more similar materials, deriving useful energy from the fission of lithium, making a metamaterial with a higher Young's modulus than diamond with negative-elasticity inclusions, constructing logic gates out of fluid vortices, building a usable hypersonic plasmoid pistol (one that doesn't require explosive flux compression pumping!), building transistors that function at 800 degrees, finding a SHS route to cubic boron nitride, making a 50-tesla magnetic field in the lab, finding a way to construct quantum-dot solar panels that's cheap and scalable enough to undercut poly-Si, confining a particle of gold in a stable minimum of the Casimir potential, or finding a way to fabricate high-quality optics apparatus rapidly out of aluminum foil; but the obstacles to these problems are (or were) mostly not that we don't have any useful hypotheses to test.

I’ve switched fields a couple times. Figure out how to provide value in the field you want to be in and start practicing it. Once you figure out what that is start looking for opportunities (job openings, contractors needed, etc). It’s really hard to find people that know what they’re doing; so if you know how to do something and can show it, it won’t matter that you don’t have formal education.

No worries!