Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

101–110 of 169 posts

Re: Why isn't preprint review being adopted?

#101

Post by rhelz »

sseagull wrote: post #95
michaelt wrote: post #33

Earlier quoted context omitted.

> The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.

In what area of science would it take only a week or two to replicate? It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top…

Science is very deep; I’m sure that to replicate some studies it would cost as much as performing the original study did.

But whether it’s $10k or $100k, we really should provide the funds to do it. Expensive? Yeah, but not as expensive funding grants for generations of psychology professors and getting nothing—or worse than nothing—in return.

Psychology could fix its replication crisis tomorrow if as part of writing every grant, they also calculated what it would take for another group to replicate their experiments, and put that money in escrow to hire reviewers and replicators who had to sign off on any papers published.

Re: Why isn't preprint review being adopted?

#102

Post by bumby »

rhelz wrote: post #99
bumby wrote: post #84

Earlier quoted context omitted.

> If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.

> If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.

Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.

Re: Why isn't preprint review being adopted?

#103

Post by ramblenode »

timr wrote: post #73
ramblenode wrote: post #49

Earlier quoted context omitted.

> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about t…

> A power analysis is how you determine reproducibility All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. > False positives produce inflated effect sizes Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's…

> All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive.

This is true when we are dealing with an uninformative prior, but published research is known to be biased toward positive results and uncorrected multiple comparisons. This situation leads to small sample studies with high random variance being paradoxically correlated with significant results. High random variance appears as a false large effect size in the published result, so if the power is low when calculated with a smaller (adjusted) effect, there is reason to believe that the p-value is inflated. See e.g. Andrew Gelman's work on small sample studies, garden of forking paths or [0].

> Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator.

Exactly! The implication being the above.

[0] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

Re: Why isn't preprint review being adopted?

#104

Post by rhelz »

matthewdgreen wrote: post #94
hiddencost wrote: post #37

Earlier quoted context omitted.

I hear this sentiment a lot. There was a time when academia was intensely driven by culture. People did stuff because they cared about the community surviving. It is, in fact, possible to choose the "cooperate" triangle of the prisoner's dilemma, but it takes a lot of work, and is vulnerable to being wrecked by people who don't care / believe in the goals.

If you keep increasing publication requirements in a Red Queen’s race, you both increase the amount of peer-review required and also reduce the time that qualified scientists have to do it. No matter how much people want to build a functioning system, they only have so many hours in the day.

A valid concern, but what is driving this red-queen race is that you are competing with people who cheat. If you’ve got one really solid paper published, but your competitor fraudulently published 10, it’s real tempting to fraud a few papers yourself.

But they multiply fraudulent papers because they can get away with it, and they can get away with it because nobody is really reviewing or replicating those results.

I propose a fix: if a paper wouldn’t get published unless somebody has replicated the results—-there would be a lot less papers published, and the the expectation of how many papers a prof should publish a year would fall to reasonable levels.

It’s the fraud which is killing us. Get rid of the fraud and a lot of the other problems with academia would start resolving themselves.

Re: Why isn't preprint review being adopted?

#105

Post by rhelz »

bumby wrote: post #102
rhelz wrote: post #99

Earlier quoted context omitted.

> If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.

Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.

I propose to supply those incentives—-pay for good reviewer and replicators.

If you want something done, you gotta pay for it. We can’t just rely on volunteers.

Re: Why isn't preprint review being adopted?

#106

Post by anticensor »

Muller20 wrote: post #11
s0rce wrote: post #7

Earlier quoted context omitted.

While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.

Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, wh…

> Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work.

Oh you mean those phantom "off topic"/"out of scope" reviews.

Re: Why isn't preprint review being adopted?

#107

Post by anticensor »

YeGoblynQueenne wrote: post #65
pacbard wrote: post #9

Earlier quoted context omitted.

To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either…

>> (Light, Singer and Willett, 1990) A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?

Hacker News does not have a LaTeX compiler.

Re: Why isn't preprint review being adopted?

#108

Post by abhgh »

currymj wrote: post #71
heikkilevanto wrote: post #62

Earlier quoted context omitted.

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.

Unfortunately the state of meta-review is similar to that of reviews. Rarely delves deeper, mostly acts as a summarizer for the independent reviews.

Re: Why isn't preprint review being adopted?

#109

Post by abhgh »

kelipso wrote: post #91
SkyPuncher wrote: post #57

Earlier quoted context omitted.

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles: * People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results. * People who aren't specialists and will need to corroborate evidence a…

> run experimentation themselves to validate results Let's just breeze through an that like it's nothing haha. Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience. There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper…

Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.

Re: Why isn't preprint review being adopted?

#110

Post by ccppurcell »

YeGoblynQueenne wrote: post #60
ccppurcell wrote: post #44

Peer review is pretty unpopular round these parts. In mathematics/TCS I've had mostly good experiences. Actually most of the time the review process improved my papers. Clearly something is rotten about the way peer review is implemented in the empirical sciences. But think of all those high profile retractions you read about these days. Usually that comes about by a sort of post hoc peer review, not by anything rese…

Not to be mean to the HN community but at least a substantial minority of people who complain about peer review on here have no experience of peer review, even in applied CS and AI and machine learning, which are the hot topics today. But they've read that peer review is broken and, by god, they'll let the world know! For science! I publish in machine learning and my experience is the same as yours: reviews have main…

Agreed wholeheartedly but didn't want to come out swinging!

Not only that but my experience of reviewing has also been positive, and has given me ideas for research and how to present research (notation, paper structure etc)

Except my first review which was a 100 page survey paper on a very specific kind of inequality that exists for practically any graph invariant, so every page was pretty much identical just with alpha then beta then omega then chi... And the deadline was my birthday!

Post reply on HN