Slowly, all articles published in this magazine are making their way to the front page.
Rebuilding after the replication crisis
31–40 of 76 posts
Re: Rebuilding after the replication crisis
#32>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…
What does money matter once a study is already done? Finding the right incentive balance between novelty and rigor isn't trivial, I get the debate surrounding grant funding. But there should be 0 debate from the perspective of what is publishable, given that it's already done. My guess is part of the resistance is fear from established PIs that their work won't actually replicate. Even if a minority of profs, I suspe…
Journals want highly-cited papers because that boosts their impact factor, makes them more popular and therefore makes them, yes, money.
A paper claiming a new result generally gets more citations than a paper saying "we replicated the study of Doe et al. and reproduced the results", even if the latter is equally or more useful from a scientific standpoint.
Re: Rebuilding after the replication crisis
#33How could any replication study sample from the same pattern of people, given that there are so many cultures with different ways of living, eating, spending time etc. All of these cultures also constantly change. Universal representativity is unattainable, so any replication will have a number of unknown sample variables that changed. This is still not highlighted enough in most papers I read.
That's the whole point of sampling. If you get a big enough sample, and you have a reasonably fair way of getting it, then all those things wash out as individual variation.
I agree. But it is very rare that I see a study that samples people living in more than one country, not to speak of sampling of people from all countries in the world. My point is this: If a study samples only people from the US, it should be highlighted very early (like in the first sentence) that it is only valid(ated) in the US and cannot be applied elsewhere, without testing.
Re: Rebuilding after the replication crisis
#34Earlier quoted context omitted.
What does money matter once a study is already done? Finding the right incentive balance between novelty and rigor isn't trivial, I get the debate surrounding grant funding. But there should be 0 debate from the perspective of what is publishable, given that it's already done. My guess is part of the resistance is fear from established PIs that their work won't actually replicate. Even if a minority of profs, I suspe…
Journals want highly-cited papers because that boosts their impact factor, makes them more popular and therefore makes them, yes, money. A paper claiming a new result generally gets more citations than a paper saying "we replicated the study of Doe et al. and reproduced the results", even if the latter is equally or more useful from a scientific standpoint.
And it's much more deeply rooted in human nature than merely the design of peer review or the quantitative metrics used for evaluating scientists.
Indeed, it is not just about money either. Intangible prestige and status among the scientific expert community is just as much, or even more, coveted. Do people read your papers and talk about it at dinner parties? Do you get invited to give talks at prestigious institutions? Do a lot of interesting and similarly active people turn out to your talks? Do people with good connections and resources want to collaborate with you on exciting ideas?
And it turns out that what people including scientists actually care about is novel, bold, visionary ideas, not drone-like repetitive meticulous detail-oriented work following the footsteps of some other group. People want something new, something cool, something flashy, something sexy, something surprising. Not just the media! Scientists themselves, too!
Re: Rebuilding after the replication crisis
#35>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…
It probably takes a different type of academic to do replication studies. It's a specific kind of detective work, to see if you can expose someone else's mistakes which are sometimes even deliberate. We need more of this type of people. EDIT: In addition to this, perhaps we should stimulate new students to do a replication study as part of their education.
It takes the sort of person who doesn't care about collecting a bunch of enemies, possibly well-connected ones.
Re: Rebuilding after the replication crisis
#36>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…
not only social sciences. Except for the very visible areas of ML, the same happens in actual science and engineering... Example: the thousands of fraudulent XRD spectra of made-up compounds.
Even in ML, it's common knowledge that the long tail of papers demonstrate brittle effects that don't really replicate/generalize and often do uncomparable evaluations, fiddle with hyperparameters to fit the test data, use various evaluation tricks (Goodhart's Law) to improve the metrics, sometimes don't cite better prior work, etc. etc. Industry people definitely know not to just take a random ML paper and believe that it has any use for applications.
This isn't to say there are no good works, but in a field that produces >10,000 papers per year, the bulk of it can't be all that great, but academics have to keep their jobs, PhD students have to graduate etc. So everyone keeps pretending.
Re: Rebuilding after the replication crisis
#37Science is an approach to epistemology.
My conjecture is that all truths must be experienced. This aligns with the notion of “nullius in verba”, the original motto of the Royal Society, arguably the birthplace of modern science.
Science takes place in a laboratory. Whether or not ink on a page is true depends on nothing other than replicating the methods for oneself.
That the current environment is for printing ink on paper and calling it a day tells me that we’ve moved on from science as an epistemological solution to the notion of truth and regressed to an era of truth emanating from privileged authorities.
Re: Rebuilding after the replication crisis
#38>One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). A while back, I asked a psychology professor why replication studies were frowned upon. She said something like "studies need to be uni…
not only social sciences. Except for the very visible areas of ML, the same happens in actual science and engineering... Example: the thousands of fraudulent XRD spectra of made-up compounds.
> Example: the thousands of fraudulent XRD spectra of made-up compounds.
Interesting, didn't hear about this case before. Can you provide a link?
Re: Rebuilding after the replication crisis
#39Statistical research based psychology is hardly a real science. It has no feedback from reality. It has nothing whose success in the real world depends on the accuracy of the research. Clinical psychology is still useful in my opinion. It still helps people. There's a real feedback loop where understanding can change outcomes. I dislike the falsifiability approach (if its falsifiable it's science) and the peer review…
In fundamental research you don't typically know the potential applications where it will be useful. In fact I dislike the current overfocus on applicability, your research can't get media-boosted unless you somehow say it's a step in curing cancer or solving climate change. Similarly, pure math is valuable even without foreseeable uses (if pressed, they will say cryptography etc. but they shouldn't have to). Curiosi…
It's not the real world applicability that I care about, it's about verifying your understanding beyond just passive observation. It's very easy to give the wrong story about something true, but much harder to use wrong story to build something.
I hardly care if nobody will gain anything from what you built - I only care that in building it, you proved your understanding. It doesn't even have to be building something useful. Even mathematical proofs count. It wasn't enough to hold a nice story in your head about the behavior of mathematical objects to get a proof - you had to use that understanding to write the mathematical proof. You did something that would fail if your understanding was incorrect - and everyone can objectively judge your success.
Re: Rebuilding after the replication crisis
#40Earlier quoted context omitted.
Physicists still study aether, but that doesn't mean it's taken as fact. Understanding where ideas and notation came from and how a field has grown to it's current state is important.
Neither psychologists nor physicists study Freud or aether.
I am a practicing physicist and I took a whole course on the history and development of physics and the natural sciences in general. It was useful.
For example, the general public, and even scientists and engineers, still talk about "heat flow" as though temperature itself is a fluid being transfered between objects. This is incorrect physically, but nevertheless there are clear mathematical analogies between how objects in contact with each reach equilibrium, and other physical systems, like two containers filled with water to different levels and connected with a pipe at the base. The reason for this is entirely historical, and if one is not mindful of that the terminology can be very misleading.