Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Published Research Findings Are False (2005)

fermatslibrary.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Why Most Published Research Findings Are False (2005)

#11

Post by TeMPOraL »

reasonattlm wrote: post #4

This is why one has to read papers in context. You can't just read a paper: you have to then go off and read some reviews that cover the surrounding science. Then read some other papers on the topic of interest. Pay attention to dates on papers, as opinions change with time. Note the disagreements. Reading the scientific literature is reading the moment to moment output of a noisy search algorithm. Any given publicat…

An interesting perspective, but it doesn't really dissolve the big problems - that the "noise" in the output of this "search algorithm" is absurdly high, and that single papers often drive real-world decisionmaking.

Re: Why Most Published Research Findings Are False (2005)

#12

Post by tensor »

It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences.

It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." There is definitely a problem with accuracy in many scientific fields that needs to be addressed, but the baby doesn't need to be thrown out with the bath water.

Re: Why Most Published Research Findings Are False (2005)

#13

Post by stinos »

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

The first paragraph reads a bit like a possibly faulty generalization based on your personal experience as covered by the rest of the post. I might be wrong, but I don't think it's that bad. Depends on the field maybe, and I don't read enough papers myself to have a proper complete view on this, but I definitely recall reading papers which where about recreating another experiment, figuring out that it was correct and adding some more steps to gain additional insight.

Re: Why Most Published Research Findings Are False (2005)

#15

Post by bad_user »

tensor wrote: post #12

It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." Th…

While I agree with the general sentiment that not all science is equal, it’s important to remember that math isn’t science.

Science is empirical and involves usage of the _scientific method_ for acquisition of knowledge and math is disqualified.

Of course math sits at the foundation of the sciences we are talking about. And truly _proving_ something involves math. That’s not what we are talking about though.

And computer science is just like math. It’s not empirical, it’s not characterized by the scientific method. It’s not science either.

Re: Why Most Published Research Findings Are False (2005)

#16

Post by epistasis »

tensor wrote: post #12

It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." Th…

It's not even talking about biology research, or even most medically related research.

While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.

Re: Why Most Published Research Findings Are False (2005)

#17

Post by goatlover »

reasonattlm wrote: post #4

This is why one has to read papers in context. You can't just read a paper: you have to then go off and read some reviews that cover the surrounding science. Then read some other papers on the topic of interest. Pay attention to dates on papers, as opinions change with time. Note the disagreements. Reading the scientific literature is reading the moment to moment output of a noisy search algorithm. Any given publicat…

> The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape.

Good point, but the problem is the media and public opinion often don't recognize this. So some research result that gains notoriety is treated as the truth from the mountain top without this context.

Re: Why Most Published Research Findings Are False (2005)

#18

Post by serghiou »

epistasis wrote: post #16
tensor wrote: post #12

It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." Th…

It's not even talking about biology research, or even most medically related research. While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.

Good point - the problem is that the grand majority of researchers publish only a very small part of what they actually did - this creates a missing data problem and greatly skews perceptions of those reading their work. If everyone published everything they had done, the problem with reproducibility would have been greatly reduced.

Re: Why Most Published Research Findings Are False (2005)

#19

Post by serghiou »

Super important paper, the implications of which have been corroborated repeatedly within the most prestigious publications in psychology, social sciences/economics and cancer biology. If you'd like to read more about such issues and actually work on doing something about it, you may want to check out the Reddit community (https://www.reddit.com/r/metaresearch/) and a recent initiative at Stanford (http://reproduciblescience.stanford.edu/).

Re: Why Most Published Research Findings Are False (2005)

#20

Post by raarts »

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

And me thinking that science isn't about trying to confirm findings, but trying to disprove them and if nobody can do that, the original hypothesis must be true.
Post reply on HN