Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rebuilding after the replication crisis

asteriskmag.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Rebuilding after the replication crisis

#61

Post by thelamest »

pa7x1 wrote: post #49

We often hear these days from science commentators if Physics should receive or not funding for the next big accelerator, High Energy Particle physics detector or gravitational wave interferometer. Most frequently brought argument is the lack of interesting findings during the last decades, and if that money would be better spent in other fields that have a higher return on investment given they are more prolific in…

Finding new particles and studying astronomical events has a larger nugget of similarity with the psychology modus operandi than you may think. They’re studying rare or unique objects with extremely finicky measurement tools; enormous amounts of noise are statistically treated to extract a signal. Victor Ninov claimed to have discovered new elements, causing years of disappointing failures to replicate (https://en.wikipedia.org/wiki/Victor_Ninov). Even if the actual scientists learn their lesson, with some regularity popular science media announce that “the Standard Model just got overturned” or that this or that new quark was discovered.

Re: Rebuilding after the replication crisis

#62

Post by caddemon »

Al-Khwarizmi wrote: post #32
caddemon wrote: post #18

Earlier quoted context omitted.

What does money matter once a study is already done? Finding the right incentive balance between novelty and rigor isn't trivial, I get the debate surrounding grant funding. But there should be 0 debate from the perspective of what is publishable, given that it's already done. My guess is part of the resistance is fear from established PIs that their work won't actually replicate. Even if a minority of profs, I suspe…

Journals want highly-cited papers because that boosts their impact factor, makes them more popular and therefore makes them, yes, money. A paper claiming a new result generally gets more citations than a paper saying "we replicated the study of Doe et al. and reproduced the results", even if the latter is equally or more useful from a scientific standpoint.

I think this is less clearly true than the funding hype imbalance though. I've seen PIs complain that a meta analysis took citations away from their paper that was included in said meta analysis. A replication study that is well written, particularly if it synthesizes replication results of multiple studies, could overtake original work in citations over time IMO. Doubly so if published in a journal that is already fairly reputable.

Re: Rebuilding after the replication crisis

#63

Post by caddemon »

bonoboTP wrote: post #34
Al-Khwarizmi wrote: post #32

Earlier quoted context omitted.

Journals want highly-cited papers because that boosts their impact factor, makes them more popular and therefore makes them, yes, money. A paper claiming a new result generally gets more citations than a paper saying "we replicated the study of Doe et al. and reproduced the results", even if the latter is equally or more useful from a scientific standpoint.

And it's much more deeply rooted in human nature than merely the design of peer review or the quantitative metrics used for evaluating scientists. Indeed, it is not just about money either. Intangible prestige and status among the scientific expert community is just as much, or even more, coveted. Do people read your papers and talk about it at dinner parties? Do you get invited to give talks at prestigious instituti…

Most scientists only want something flashy within the confines of the "cookbook" they've already learned on how to do studies. NIH grant applications are formulaic as hell.

Moreover, many labs organize around a methodology like fMRI rather than around any particular type of question. Imagine planning a multi-decade future career around a single tool that exists today and then pretending you truly care about novelty.

You're spot on that people care deeply about academic status. But what is valued in gaining that status has become deeply broken. The fastest way to status is to put out multiple overhyped individual publications that meet the minimum viable novelty threshold for inclusion in a good journal. Half of the battle is a marketing game.

Increased rigor is a time and money commitment that many aren't willing to make, but true novelty is a much bigger risk and a good way to kill a career for anyone not yet tenured.

Re: Rebuilding after the replication crisis

#64

Post by bakuninsbart »

amelius wrote: post #30
portpecos wrote: post #7

>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…

It probably takes a different type of academic to do replication studies. It's a specific kind of detective work, to see if you can expose someone else's mistakes which are sometimes even deliberate. We need more of this type of people. EDIT: In addition to this, perhaps we should stimulate new students to do a replication study as part of their education.

Imo. this would actually be a great target of public funding. A large international research organization with the the sole purpose of replicating, validating and criticising existing research.

As you correctly point out, the skillset is slightly different to doing novel research, the attitude is different as well, so it would make sense to pool talent. Since there is no money to be made directly with it, governments should step in. And to avoid the problem of journals not wanting to publish replications, the institute would have to be well funded enough to self-pubpish with prestige.

Re: Rebuilding after the replication crisis

#65

Post by williamcotton »

krapp wrote: post #60
williamcotton wrote: post #56

Earlier quoted context omitted.

My point is that we’re probably no longer in what future historians will consider the scientific era. “Believe in science” would be an incoherent statement to Newton.

>“Believe in science” would be an incoherent statement to Newton. It wouldn't, because Newton considered himself primarily an alchemist and philosopher, and viewed all of his "scientific" endeavors as a means to the end of understanding God through understanding Creation.

Really, the guy who was president of a club whose motto is “take no one’s word for it” would be fine with just like, taking someone’s word for it?

Re: Rebuilding after the replication crisis

#66

Post by nathanaldensr »

portpecos wrote: post #7

>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…

With so many of these stories, is it any wonder that people now refer to this as Science™?

Re: Rebuilding after the replication crisis

#67

Post by bonoboTP »

caddemon wrote: post #63
bonoboTP wrote: post #34

Earlier quoted context omitted.

And it's much more deeply rooted in human nature than merely the design of peer review or the quantitative metrics used for evaluating scientists. Indeed, it is not just about money either. Intangible prestige and status among the scientific expert community is just as much, or even more, coveted. Do people read your papers and talk about it at dinner parties? Do you get invited to give talks at prestigious instituti…

Most scientists only want something flashy within the confines of the "cookbook" they've already learned on how to do studies. NIH grant applications are formulaic as hell. Moreover, many labs organize around a methodology like fMRI rather than around any particular type of question. Imagine planning a multi-decade future career around a single tool that exists today and then pretending you truly care about novelty.…

My larger point is that this is a general human problem, not a science-specific one. Voters listen to the flashy corrupt demagogue politician who speaks to emotions, not the boring one who speaks in nuance and works transparently. In dating people complain about the other gender being shallow and overlooking deeper values. On TV, people watch garbage reality shows so those make the most money. On YouTube, people click thumbnails with obnoxious facial expressions so those win out. In the movies the safe bet is to churn out films from the same franchises as before (not unlike the scientist who barely changes things between papers).

It's not gonna change, one has to learn to accept it and to adapt to it.

Re: Rebuilding after the replication crisis

#68

Post by mhuffman »

bonoboTP wrote: post #35
amelius wrote: post #30

Earlier quoted context omitted.

It probably takes a different type of academic to do replication studies. It's a specific kind of detective work, to see if you can expose someone else's mistakes which are sometimes even deliberate. We need more of this type of people. EDIT: In addition to this, perhaps we should stimulate new students to do a replication study as part of their education.

It takes the sort of person who doesn't care about collecting a bunch of enemies, possibly well-connected ones.

Also a person that doesn't care about ever being published in "approved" journals, magazines, etc.

Re: Rebuilding after the replication crisis

#69

Post by mhuffman »

3pt14159 wrote: post #48
mattkrause wrote: post #45

Earlier quoted context omitted.

That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested. The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtr…

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives ar…

>Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives are on the line if you get it wrong, so people actually do the work to make the testing gear and align the incentives that need to be aligned.

You don't think lives are on the line with psychology? The replication crisis there should be terrifying but because of the extra degree of separation between theory and result, it is, for some reason, just hand-waved over.

You are right that if people died because of a structural engineering mistake due to incorrect information people would be sued and possibly sent to prison. In mental health, however, people can die in droves and nothing! Or public policy that destroys the lives of millions can be made on bad research and just shrugged off later.

Re: Rebuilding after the replication crisis

#70

Post by krapp »

williamcotton wrote: post #65
krapp wrote: post #60

Earlier quoted context omitted.

>“Believe in science” would be an incoherent statement to Newton. It wouldn't, because Newton considered himself primarily an alchemist and philosopher, and viewed all of his "scientific" endeavors as a means to the end of understanding God through understanding Creation.

Really, the guy who was president of a club whose motto is “take no one’s word for it” would be fine with just like, taking someone’s word for it?

If that "someone" was the Bible, yes. Apart from his occult work (which like most Western magic was done in a heretical Christian context) he published numerous works on Biblical literalism and prophecy. That Newton considered his work primarily an expression of his religious views isn't exactly a secret.

You made the mistake of reaching too far back in time, searching for an example of someone who should have been aghast at the premise of applying faith to science, and found someone for whom they were one and the same. Although he did reject a lot of the orthodox views of the church, he certainly didn't reject religion outright for its lack of verifiability.

Post reply on HN