This is especially true in math and physics. You will find that no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial. Like, what about analogous of elliptic functions for non-elliptic integrals? already been done. The past century has seen a huge explosion of research into STEM subjects, from m…
My guess is this slowdown is happening for a few reasons:
1. Increased number of researchers = increased team size = less innovation. The article shows increase in team size but doesn't ponder the implications. Teams shy away from bold ideas, in my experience. If you want innovation it has to come from individuals empowered to work alone and recruit slowly. The moment you're put in a team situation you are suddenly expected to pitch and convince others to take a risk on an idea that perhaps you aren't even sure about yourself yet, which is a high bar to meet. And the team won't want to try it because if it works the glory will associate with the individual who came up with the idea and drove it forwards, leaving the others in the shade. So teamwork puts pressure on people to propose 'safe' ideas that were found outside the group, which nobody will object to and which everyone can share equally.
NB: non tech firms struggle to create new tech partly for this reason. They have a culture of creating so-called innovation teams. This practice is rampant in finance for example. I never saw an innovation team do anything truly surprising. You could always guess up front what topics they'd be "researching" before learning anything about them because the range of topics was so narrow.
2. State subsidies. We know these kill worker efficiency. If that weren't true the USSR would never have fallen behind the USA in terms of wealth. What the article refers to as science is really academia, and academia is dominated by ever increasing amounts of government money. Whilst the article phrases this as science getting "harder" it can also be seen as researchers simply becoming less efficient than they were in the past, which is exactly what we'd expect to happen given that academia is a parallel planned economy. Efficient here means in terms of discovery production not paper production, of course.
3. Falling paper quality. Another way to view (2). I feel like half my HN comments are about this problem these days but the quality of papers in some research fields is staggeringly low, sometimes junk quality. As in, you could throw out 90%+ of the papers and the field would get better not worse. In a few fields like "social bot" research or epidemiology I'd struggle to name any recent papers that weren't intellectually fraudulent in some way. If you join a field as a researcher because you feel like it's an important topic, and then discover that the papers published in the last 10-20 years are much more likely to be non-replicable or have nonsense methodologies than those published 50 years ago, then you'll probably end up reading older papers because you feel you get more out of them. Then you'll end up citing them more often as a result.
I definitely feel I saw this when reading the epidemiology literature. Papers from the 1950-1990 period were quite different to modern papers. Way less fancy maths, much easier to read, more obvious and logical questions being asked and no WTF moments. You definitely got a feeling that the authors were intellectually curious and wanted to understand epidemics. From 2000 onwards the papers became nearly always useless.
A big part of this is the post-2000s era explosion in the use of advanced statistical methods and, especially, the acceptance of unvalidated models as "science". The creation of free tools like R and STAN made it much easier and so many papers now are just people playing around with R and random arbitrary datasets. They plot some regressions and publish a paper. Unvalidated modelling seems to have destroyed a lot of fields, because to people who are rewarded for publishing it's basically crack cocaine. If your papers have to describe factual things about reality then you're limited in how much you can write about by cost of experimentation, difficulty of discovering new things etc. If your papers can describe arbitrary scenarios invented in a computer with no care given to validity, then those caps are removed and you can publish an infinite number of papers. Yet the actual value of such papers can be zero or lower.
So yeah. He shows graphs of discovery fall in exactly the same way across every single field he looks at, whilst output grows exponentially, and then concludes that "science" in general is getting harder. But then that's clearly not been true of some fields like AI lately. It seems more likely that such a strongly correlated trend is caused by structural issues in academia rather than a true general effect.