Live data from Hacker News

Most Published Research Findings Are False (2005)

ncbi.nlm.nih.gov

31–40 of 41 posts

Re: Most Published Research Findings Are False (2005)

#31

Earlier quoted context omitted.

Can you elaborate as to why? Snark is not really the goal here.

It's a perfectly valid, and indeed important question. Self-application is an important form of scientific methodology. Think e.g. of programming languages development: one of the first things you do is bootstrap a compiler.

You're conflating science publishing philosophy with standard software development?

Re: Most Published Research Findings Are False (2005)

#32
Most code is buggy despite passing all tests.

Publication is a sign of persistence, plausibility and a minimal quality check. Peer review improves articles, it does not guarantee correctness.

That's also why one should be extremely careful about things that are not peer reviewed. Not because the bar is so high, but because it's so low.

Re: Most Published Research Findings Are False (2005)

#33
post #32

Most code is buggy despite passing all tests. Publication is a sign of persistence, plausibility and a minimal quality check. Peer review improves articles, it does not guarantee correctness. That's also why one should be extremely careful about things that are not peer reviewed. Not because the bar is so high, but because it's so low.

Just because the bar is low doesn't mean its height is in any way related to its discernment of quality. Say, for example, the bar was simply based on the institution that the author is employed by. That may be a low bar in that anyone from Harvard can get a paper published, but it also doesn't actually tell us anything about the paper's validity. Whereas I wouldn't be skeptical of a good Rush University paper, simply because its authors weren't prestigious enough for Nature to pay attention to.

Re: Most Published Research Findings Are False (2005)

#34
post #29

I learned this in middle school during the Science Fair. My experiment didn't work. Up against the deadline and not wanting a 0, I simply made up the data. I got a nice big A. Probably exactly what happens in the real world too.

What was your experiment?

Re: Most Published Research Findings Are False (2005)

#36
post #33
post #32

Most code is buggy despite passing all tests. Publication is a sign of persistence, plausibility and a minimal quality check. Peer review improves articles, it does not guarantee correctness. That's also why one should be extremely careful about things that are not peer reviewed. Not because the bar is so high, but because it's so low.

Just because the bar is low doesn't mean its height is in any way related to its discernment of quality. Say, for example, the bar was simply based on the institution that the author is employed by. That may be a low bar in that anyone from Harvard can get a paper published, but it also doesn't actually tell us anything about the paper's validity. Whereas I wouldn't be skeptical of a good Rush University paper, simpl…

I was talking about getting your paper peer reviewed, not placing it in a particular journal. If you want to get into PLOS One I don't think your university matters very much.

Re: Most Published Research Findings Are False (2005)

#37
post #29

I learned this in middle school during the Science Fair. My experiment didn't work. Up against the deadline and not wanting a 0, I simply made up the data. I got a nice big A. Probably exactly what happens in the real world too.

If you would have gotten a 0 then your science teachers failed you I'd say. A failed experiment is just as valid a result as a successful one if the goal is to gather data.

Re: Most Published Research Findings Are False (2005)

#38

I think the solution to this is that science publication needs to be done like science itself. Just as scientists commit to doing experiments without knowing what the results of the experiments will be, publishers need to commit to publishing experiments without knowing what the results of the experiments will be. Publishers need to be involved earlier in the process. Instead of submitting a completed research findin…

[deleted]

Re: Most Published Research Findings Are False (2005)

#39

Earlier quoted context omitted.

It's a perfectly valid, and indeed important question. Self-application is an important form of scientific methodology. Think e.g. of programming languages development: one of the first things you do is bootstrap a compiler.

You're conflating science publishing philosophy with standard software development?

Programming language development is not standard software development.

PL research is a discipline with no really feasible empirical methods for robust and replicable evaluation of programming language ideas. Using self-application is a test of internal coherence -- if your new language is not good at writing a compiler, then maybe it isn't such a good idea.

Self-application is a well-known scientific methodology. Would you trust a physical theory that predicted the impossibility of physicists?

Re: Most Published Research Findings Are False (2005)

#40
Inside the Fake Science Factory (2018) is an amazing DEFCON talk, by 3 people who stumbled on and investigated an unbelievably huge worldwide network (400,000 scientists involved) of fake-scientific conferences, journals, websites raking in money and enabling people to make claims that their products are scientifically proven. The presenters got hackers to track down the people behind one of the largest organizations, and analyze the huge numbers of conferences/papers. They present/publish nonsensical talks on graphs and bees+cancer to see what peer review there is... A must watch. Very funny, if depressing.

https://www.youtube.com/watch?v=ras_VYgA77Q

Post reply on HN