Early 20th century has seen a dramatic change in universities: from teaching institutions, they became scientific ones. PHD programs were created in 19th century Germany, but became a requirement for professorship in most universities by the 20th century. The creation of these programs meant that being a scientist became a real job, and we began devoting resources to make sure the brightest minds could devote to scientific pursuit. This might have been a major reason of the scientific innovations that predates this "mini dark age".
To answer the question about the causes of this "mini Dark Age", I suggest two points to explain what has changed:
1) The transformation of the universities into research facilities has hit the moment of diminishing return. We scientists are now hyper-specialized and it is really hard to grasp even a tiny fraction of our field, whereas early 20th century scientists were expected to know maybe 10% of their discipline.
As sciences stabilize, great innovations become rarer and harder to achieve, and small incremental discoveries become the norm. The "publish or perish" environment makes scientists very cautious about their statements, non-working experiments are not published, preventing colleagues to avoid the same mistakes that must be made again and again.
2) The 70s, the decade associated with science slowing its innovation pace coincides with the liberalization of the financial market, known as neoliberalism. Welfare state is then discredited and privatization is then seen as the way to offer services. Banks, hedge funds and credit agencies are now, to put it (too) bluntly, ruling the world. This system of regulation of collective processes through finance is the world we know today and drives us toward a ever increasing productivity. It also creates an imperative for profits: some like it, some don't. But major innovation is not necessary to reap profits, like science was necessary to maintain the domination of some states over others. Major corporations do prefer small incremental changes. Heard about this new thing, iPhone 7? Not a big innovation, but perfect to turn steady quarterly profits. The way big oil consortia are now panicking even show that sometimes, innovation kills profit.
Today, scholarly grants are highly dependent on industrial partnerships, which foster mostly new uses of old tech. It is much a safer investment than trying to make fundamental changes. The productivity imperative, first implemented in factories, is now imposed on universities.
*
One of today biggest hype in tech, deep learning, was created (partly) because Y. Bengio from Université de Montréal chose to spit on big corporate money to focus on fundamental research. Another great scientist, Peter Higgs (Higgs like in Higgs boson!) said that if he had to comply to the productivity required of today's young scientists, he would never had been able to think what is driving today's physics.
Both of them had to go against the flow, but how many promising scientists chose the stable, profitable, corporate career path? You guys know more than me.