Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Peer review: the end of an error?

the-tls.co.uk

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Peer review: the end of an error?

#51

Post by squiggleblaz »

improbable22 wrote: post #42
aldoushuxley001 wrote: post #25

Earlier quoted context omitted.

"Only interesting problems/solutions/results get published" I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.

Isn't this the point of journals? They communicate new knowledge, which changes the status of what is known. There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high. Things which stay constant are the job of standards bodies, and textbook write…

I don't think so. If you work in a hard science like medicine or psychology, where problems are still so hard we can't produce clear answers as to results of a paper, we shouldn't really be trusting papers until they've been reproduced by some unrelated team. But if these results don't contradict the previous papers, they're not really communicating new knowledge.

Sometimes this isn't a big deal (medicine is as far as I know regulated enough that reproduction is a part of the problem), but most science is characterised by not enough money distributed without caring about scientific outcomes.

So we do have a problem at the moment. A team who does important but thankless scientific work can't get published. Too much science is just first draft status. And the only way to determine that is that people keep trying and trying to build on it and eventually decades later the tower of cards collapses when clever new PhD students start down new research avenues.

Re: Peer review: the end of an error?

#52

Post by tzahola »

pmiller2 wrote: post #45
jhbadger wrote: post #44

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

I disagree. Even with the current peer review system in mathematics, false proofs have made it into the literature. Who knows how many more would creep in if we did away with peer review?

One can argue that false proofs made it into the literature because people trusted the reviewers - compare with: appeal to authority.

Re: Peer review: the end of an error?

#53

Post by tincholio »

pizlonator wrote: post #50

I've published in CS conferences, CS journals, and on a blog. Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you. OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for confer…

The difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it here, you will likely get many eyeballs. If you post serious science, you'll get much less feedback, if any.

Re: Peer review: the end of an error?

#54

Post by wbillingsley »

In some fields, such as HCI (and research through design) I think there could be value in moving to a system where instead you version papers and also publish the criticisms that led to its refinement.

If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do.

All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems.

Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.

Re: Peer review: the end of an error?

#55

Post by squiggleblaz »

dragonwriter wrote: post #40
arca_vorago wrote: post #39

I've said it before and I'll say it again. The focus should be on reproducing studies, not on peer-review. Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues. I used to hold the peer-review and jou…

> The focus should be on reproducing studies, not on peer-review The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.

Reproducing is a form of peer review (if you assume that a person who successfully attempts to reproduce your work is ipso facto your peer).

But the fact that your peer (i.e. another academic who works in the same area) has read your paper and commented on it in the process of publication (i.e. customary "formal peer review") does not reflect on its reproducibility. The peer, being a human, has attentional deficits and there's probably true facts that increase the chances of the result occurring by chance which are not considered worthy of publication.

Until a paper's result is reproduced, it can't be regarded as reproducible.

Re: Peer review: the end of an error?

#56

Post by dunk010 »

I am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong problem. If you want to fix the system then you have to come up with a new way of rating academic's publications, which would then be fed back into the career progression system in their individual institutions. Good luck with that.

Re: Peer review: the end of an error?

#57

Post by pizlonator »

tincholio wrote: post #53
pizlonator wrote: post #50

I've published in CS conferences, CS journals, and on a blog. Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you. OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for confer…

The difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it he…

My blog posts typically have more scientific content than a CS paper.

Re: Peer review: the end of an error?

#58

Post by iandanforth »

dunk010 wrote: post #56

I am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong pro…

While this is true in many institutions there are now useful proxies, specifically a Google Scholar profile or an individual h-index which can be calculated without publication impact factor.

You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.

Re: Peer review: the end of an error?

#59

Post by dunk010 »

iandanforth wrote: post #58
dunk010 wrote: post #56

I am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong pro…

While this is true in many institutions there are now useful proxies, specifically a Google Scholar profile or an individual h-index which can be calculated without publication impact factor. You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addi…

Of course there's an untruth in what I said. Ostensibly it's all about Impact Factor, but in reality it's all about how much money you are bringing in. Academia is in a sad state.

Re: Peer review: the end of an error?

#60

Post by wyattpeak »

danieltillett wrote: post #37

The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service. The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the m…

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard?

Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.

Post reply on HN