Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Published Research Findings Are False (2005)

fermatslibrary.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Why Most Published Research Findings Are False (2005)

#21

Post by chrisseaton »

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

I think it's pretty common to ask PhD students to reproduce some results as part of their initial literature review.

Re: Why Most Published Research Findings Are False (2005)

#22

Post by chrisseaton »

tomrod wrote: post #8
majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

Why not arxiv?

Why not arXiv... what?

Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.

Re: Why Most Published Research Findings Are False (2005)

#23

Post by duckerude »

swagatkonchada wrote: post #14

Maybe not in mathematics

>Anecdotal evidence suggests that as many as a third of all papers published in mathematical journals contain mistakes - not just minor errors, but incorrect theorems and proofs…

http://www.gwern.net/The-Existential-Risk-of-Mathematical-Er...

Re: Why Most Published Research Findings Are False (2005)

#24

Post by tomrod »

chrisseaton wrote: post #22
tomrod wrote: post #8

Earlier quoted context omitted.

Why not arxiv?

Why not arXiv... what? Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.

Why not host reproductions in arXiv to build up a name, similar to how devs use github to build a profile.

Re: Why Most Published Research Findings Are False (2005)

#25

Post by chrisseaton »

tomrod wrote: post #24
chrisseaton wrote: post #22

Earlier quoted context omitted.

Why not arXiv... what? Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.

Why not host reproductions in arXiv to build up a name, similar to how devs use github to build a profile.

The problem isn't literally hosting the reproduction study result PDF files when you're done - you could do that on GitHub or anywhere else nobody cares.

The problem is funding the time and resources needed to do the reproduction studies themselves.

> The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct.

Re: Why Most Published Research Findings Are False (2005)

#26

Post by gus_massa »

chriskanan wrote: post #7
majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication…

From your profile, you are working in Deep Learning. In other fields it's much more difficult to be sure that the replication is accurate.

Do you have the correct variety of rats? Are they receiving the same kind of food? Does they get the same illumination during the day? ... Theoretically al the details should be clear from the published paper, but most of the times the paper is full of "underspecified"[0] parts.

Also, in many fields even if the paper is only about calculations in a computer, the programs are not published (or are a mess (or an unpublished mess)) and the data are not published (or are a mess (or an unpublished mess)). So it's more difficult to even make a direct copy of the results of the paper.

Also, there is a lot of informal replication, essentially what you do but without the final publication step. Just replicate somewhat similar to the original paper, but then publish a version with some extension or tweak.

[0] aka "missing"

Re: Why Most Published Research Findings Are False (2005)

#27

Post by chrisfinazzo »

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem.

Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while.

Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.).

You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'

Re: Why Most Published Research Findings Are False (2005)

#28

Post by chrisfinazzo »

baxtr wrote: post #10
majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

Yes, that’s true. It’s also because the view is: published == valid. An idea I had once was to say: the group that repeats or falsifies the results of a paper will be published in the same journal. So this motivate people to piggy back on high profile publications. Unsure if it’ll work though

The idea that just because it was published makes it valid is insane on so many levels. Usually, it is required - or at least good sense - to address limitations and draw conclusions based on your findings.

This assertion flies in the face of that practice.

Re: Why Most Published Research Findings Are False (2005)

#29

Post by nonbel »

chrisfinazzo wrote: post #27
majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

Not surprised to hear this at all . I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spe…

>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'."

This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).

Re: Why Most Published Research Findings Are False (2005)

#30

Post by wil3 »

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

To be fair, the professors are like this because the funding agencies---ie, the government and a handful of private groups---essentially refuse to fund duplication research. I thus think that the issue has broader origins than the attitudes of individual faculty.
Post reply on HN