Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rebuilding after the replication crisis

asteriskmag.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Rebuilding after the replication crisis

#41

Post by analog31 »

machina_ex_deus wrote: post #21

Statistical research based psychology is hardly a real science. It has no feedback from reality. It has nothing whose success in the real world depends on the accuracy of the research. Clinical psychology is still useful in my opinion. It still helps people. There's a real feedback loop where understanding can change outcomes. I dislike the falsifiability approach (if its falsifiable it's science) and the peer review…

In my view, clinical psychology, like medical practice, is likely to be an art and not a science, for a long time. The reason is we can't wait for the science to come up to speed. People are suffering, and need relief, right now.

My view is that replication is just one safeguard within science, but is not the only thing needed. Confirming or refuting isolated factoids doesn't tell us much more than the factoids themselves.

I think a science needs to develop towards theories that connect those factoids into a framework. Psychology has not reached that stage yet. Compare to physics or chemistry. Some commentary has suggested that the replication crisis extends to those fields too. My graduate research project in physics was never replicated. But physics has a framework of theory that remains strong by connecting studies to one another, so that if one or more isolated studies fail on replication, the whole framework remains strong.

In some sense, "useful" could include merely "useful to science."

Re: Rebuilding after the replication crisis

#42

Post by analog31 »

roxgib wrote: post #2

Plack said, "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it ..." I wish I could be more hopeful, but it seems like a large portion of researchers in fields like psychology are too worried about their prior, poor quality research to embrace change.

Admittedly that quote is a bit of a trigger for me, since I'm an older scientist. Also, "embracing change" is a business dog whistle that is often applied when the change is onerous to the people on whom it's being imposed. Both are technically ad hominem.

A bigger problem may be that the entire body of published knowledge, and even the choice of categories that are the focus of study (such as "personality" and "intelligence") are too numerous and prevalent throughout our entire culture to readily abandon.

The right thing to do if half of your knowledge base is bunk, might be to erase all of it and start over, but that's virtually impossible.

Re: Rebuilding after the replication crisis

#43

Post by dan_mctree »

machina_ex_deus wrote: post #21

Statistical research based psychology is hardly a real science. It has no feedback from reality. It has nothing whose success in the real world depends on the accuracy of the research. Clinical psychology is still useful in my opinion. It still helps people. There's a real feedback loop where understanding can change outcomes. I dislike the falsifiability approach (if its falsifiable it's science) and the peer review…

Strongly agree. The main reason we got so much out of physics and chemistry has little to do with the statistical methods and peer review they employed, and much more with how obviously some things were repeatable when found. You need basically 100% reproduction rates to be able to do something truly useful and that's a bar you just can't hit with some fields

Re: Rebuilding after the replication crisis

#44

Post by zemvpferreira »

williamcotton wrote: post #37

Science is an approach to epistemology. My conjecture is that all truths must be experienced. This aligns with the notion of “nullius in verba”, the original motto of the Royal Society, arguably the birthplace of modern science. Science takes place in a laboratory. Whether or not ink on a page is true depends on nothing other than replicating the methods for oneself. That the current environment is for printing ink o…

Surely this is also a consequence of knowledge expansion? Up to a certain year some people could credibly be said to know everything scientific. There just wasn't much of it. I couldn't imagine verifying with my own eyes even a small subset of one field today. I doubt there's one person alive who's seen 5% of, say, all metallurgy with their own eyes.

How would we ever progress without a change in method from observation to belief?

Re: Rebuilding after the replication crisis

#45

Post by mattkrause »

bell-cot wrote: post #20

Useful background, if you aren't familiar: https://en.wikipedia.org/wiki/Replication_crisis Noteworthy is that the crisis is a huge deal in Psychology - a field which "real" scientists were sneering at a century or more ago. Personal anecdote: I once knew a guy who majored in Psychology at a pretty prestigious U.S. research university, back in the mid 1980's. He said that the Psych Dept. there did a big survey of Psy…

That sneering misses the mark, IMO.

Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested.

The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtrics account—--or a photocopier. They can often be done fairly quickly, making replications a good "warm-up" project for a new grad student. The field also seems to value replications, at least a little.

Second, psychology is actually quite hard! Some failed replications surely reflect sloppy or sketchy research practices, but many of them are probably due to reflect uncontrolled variables/variance. A physicist can produce an endless stream of photons, all of which are exact copies and totally independent of each other. Psychologists have nothing like that. People's reactions are shaped by their own idiosyncratic biases, their past history, and even what they think the experimenter wants to see. It's often not clear what variables actually matter, and in that sense, "failed" replications are sometimes more interesting than successes.

Re: Rebuilding after the replication crisis

#46

Post by mattkrause »

machina_ex_deus wrote: post #21

Statistical research based psychology is hardly a real science. It has no feedback from reality. It has nothing whose success in the real world depends on the accuracy of the research. Clinical psychology is still useful in my opinion. It still helps people. There's a real feedback loop where understanding can change outcomes. I dislike the falsifiability approach (if its falsifiable it's science) and the peer review…

That's not fair--or true.

Work on perception and memory has held up astonishingly well. Behavioral experiments from the 1800s and 1900s basically nailed down the properties of the retina, a hundred years before we had the techniques to measure it physiologically.

It's also integrated in a lot of "real world" applications: a lot of work has gone into building psychoacoustic models for audio codecs, color spaces for image reproduction, etc. Findings about attention and eye movements influence UX, and all sorts of products exploit behavioral biases (often for nefarious ends).

Re: Rebuilding after the replication crisis

#47

Post by siva7 »

analog31 wrote: post #41
machina_ex_deus wrote: post #21

Statistical research based psychology is hardly a real science. It has no feedback from reality. It has nothing whose success in the real world depends on the accuracy of the research. Clinical psychology is still useful in my opinion. It still helps people. There's a real feedback loop where understanding can change outcomes. I dislike the falsifiability approach (if its falsifiable it's science) and the peer review…

In my view, clinical psychology, like medical practice, is likely to be an art and not a science, for a long time. The reason is we can't wait for the science to come up to speed. People are suffering, and need relief, right now. My view is that replication is just one safeguard within science, but is not the only thing needed. Confirming or refuting isolated factoids doesn't tell us much more than the factoids thems…

I don't think this could work out in medical practice or psychology. There are simply too many variables that influence the outcome in those disciplines and too many unknowns how those things work in detail that can't be expressed in a formula. The brain still remains a mistery

Re: Rebuilding after the replication crisis

#48

Post by 3pt14159 »

mattkrause wrote: post #45
bell-cot wrote: post #20

Useful background, if you aren't familiar: https://en.wikipedia.org/wiki/Replication_crisis Noteworthy is that the crisis is a huge deal in Psychology - a field which "real" scientists were sneering at a century or more ago. Personal anecdote: I once knew a guy who majored in Psychology at a pretty prestigious U.S. research university, back in the mid 1980's. He said that the Psych Dept. there did a big survey of Psy…

That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested. The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtr…

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis.

In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives are on the line if you get it wrong, so people actually do the work to make the testing gear and align the incentives that need to be aligned.

Re: Rebuilding after the replication crisis

#49

Post by pa7x1 »

We often hear these days from science commentators if Physics should receive or not funding for the next big accelerator, High Energy Particle physics detector or gravitational wave interferometer. Most frequently brought argument is the lack of interesting findings during the last decades, and if that money would be better spent in other fields that have a higher return on investment given they are more prolific in their discoveries.

Guess what, it's very easy to find things when 1 in 20 of your papers can produce them by random chance (95% CI). When not outright participating in scientific fraud and p-hacking.

Perhaps the real lesson to learn from all of this is that we should defund every field that does not adhere to strict replicability controls. When funding follows findings you have set-up the perverse incentive to find something when there is nothing to get funding.

Re: Rebuilding after the replication crisis

#50

Post by MichaelZuo »

3pt14159 wrote: post #48
mattkrause wrote: post #45

Earlier quoted context omitted.

That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested. The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtr…

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives ar…

There's definitely no replication crisis in jet turbine design research also.
Post reply on HN