Scientific Regress

disturbing.

I am a post-doctoral research scientist that has wanted to be a scientist since early childhood (6-7 years old). The structure of how science is practiced seems to have changed substantially in that time frame, not for the better. Some of my childhood naivete plays a role in those perceptions, but many older scientists share that perspective.

A lot of the problems in modern science revolve around the growing focus on bringing large quantities of research funds to your institution (which gets a large cut). Getting grants has an increasingly large need for political clout due to peer-review and increased competition due to reduced funding. Consequently, labs are getting larger and more hierarchical (i.e., an increased number of long-term post-docs and research scientists). As the organizational structure of science becomes less flat, the influence of those at the top is just reinforced and tends to constrain dialogue to fit established ideas. I think that the long-term progress of science depends on placing small bets on a greater variety of ideas rather than doubling down on fewer. Unfortunately, it will always be perceived to be safer to fund conventional ideas. Peer review enforces short-term, safe bet approaches.

Large labs can churn out lots of papers, even if they are relatively financially inefficient. One way to correct this (if you agree that is a problem) might be to normalize grant scores by previous grant funding to the PI. i.e., (X papers of Y impact)/Z dollars of funding over the past 10 years. Double-blind review of grants might also help, but blinding is relatively easily circumvented.

Great article. I would say this is inaccurate though:

>"What it really means is that for each of the countless false hypo­theses that are contemplated by researchers, we accept a 5 percent chance that it will be falsely counted as true—a decision with a considerably more deleterious effect on the proportion of correct studies."

The hypotheses that the statistical "hypothesis testing" framework is usually applied to often amount to "two groups of cells/animals/people are samples from exactly the same population". Then there will be assumptions about the distribution of this population, etc. As has been noted by many (see eg Meehl 1967), such a hypothesis is pretty much always false and nothing but a strawman.

http://www.psych.umn.edu/people/meehlp/WebNEW/PUBLICATIONS/0...

I think this is largely a case of 'moloch' as described by Scott Alexander: http://slatestarcodex.com/2014/07/30/meditations-on-moloch/

In that long (but excellent essay) Scott points out that self organizing systems like human society can sometimes arrange themselves in horrible local minimas which can be very difficult to escape.

I think the way modern science is organized is decidedly suboptimal. Note that I'm talking purely sociologically: science, as a method, is still by far the best thing we've developed to understand the natural world. However, the incentives around publishing / grants / hiring work are broken in a myriad of ways, from small to massive.

For example:

- High prestige journals/conferences are basically a crapshoot: http://blog.mrtz.org/2014/12/15/the-nips-experiment.html - consider that getting a paper accepted in NIPS/published in Nature might completely change the way your career shapes up.

- Grant funding is incredibly competitive, and the way they are awarded is also a crapshot: https://psmag.com/why-the-national-institutes-of-health-shou...

- The pressure to publish/obtain grants drives people to either make up data to obtain fancy publications, or even to kill themselves: http://www.dcscience.net/2014/12/01/publish-and-perish-at-im...

- We reward researchers for brilliant clean discoveries, not brilliant methodologies. However, the outcome of a serious research project is the one thing that a scientist cannot control: science investigates the unknown, if someone explores a plausible hypothesis in a clever way, and the hypothesis turns out to be wrong, the scientist didn't do anything wrong.

- PhDs are often used as cheap labour. A very famous PI once bragged that before they got their new fancy robot, they just had 10 chinese postdocs handle all the plating.

- Private companies make an insane amount of money from publishing scientific research that's carried out with public money, and that's reviewed by scientists (largely) paid with public money who donate their money for free.

I could go on for ages - and so could almost every researcher/ex-researcher on HN. There's a few no brainer changes - but the problem is that almost everyone in power who could affect change stands to benefit from the current system.

It might be helpful to compile a list of those citations online (your own website, perhaps?). This is the sort of essay that might get shared around a whole bunch....it would be a shame for its impact to be lessened because of the inevitable judgment people will have for the hosting publication.

It may be a perfect example of the "secular bias" the publication references, but I too groaned inwardly when I went to check out the "about" section after completing my read. Not because I changed my mind on the content of the article, but because I was imagining sharing it and having to deal with an argument about publication bias (in the media) for an essay about (among other things) publication bia (in the sciences)