Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arxiv.org reaches a milestone and a reckoning

scientificamerican.com

Post reply on HN
Print view View original

71–80 of 91 posts

Re: Arxiv.org reaches a milestone and a reckoning

#71

Post by marginalia_nu »

UncleMeat wrote: post #67
marginalia_nu wrote: post #46

Earlier quoted context omitted.

> Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?

I think the core problem here is the layperson perception of peer review. People often say "it is a peer-reviewed paper!" when trying to claim that a paper is definitive and powerful. But when you speak to scientists they have a much more muted understanding of peer review. And I think that is okay. The amount of effort it would take to do a very thorough analysis of a paper is large, sometimes even approaching the c…

> In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way.

This is actually deeply problematic, as a part of what's driving the replication problems is specifically publication bias. If the only thing that gets published are unexpected or noteworthy results, you're selecting for statistical aberrations.

Re: Arxiv.org reaches a milestone and a reckoning

#72

Post by krastanov »

monkeyjoe wrote: post #57
kgwgk wrote: post #56

Earlier quoted context omitted.

> arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. I'm surprised that they no longer use the term "preprint" at all, at least it's nowhere to be found on the homepage or "about" section. The consequences of this amnesia are hilarious: https://twitter.com/gustavnilsonne/status/138948729731431219... > Why do we call it "preprints"? The term…

I think it's because the publisher retains copyright. There is a limit on how "done" the manuscript can be and still be shared for free online. Some universities have started to fight back against this by limiting the scope of copyright restrictions that publishers can impose.

No, in most of physics there is no such limit in practice. The only difference between my published work and the preprint arxiv versions is the font and whether the layout is two columns or one column. They are word-for-word identical with identical figures.

Re: Arxiv.org reaches a milestone and a reckoning

#73

Post by albertzeyer »

YeGoblynQueenne wrote: post #62
atomack wrote: post #44

Earlier quoted context omitted.

The way we used arxiv worked well in physics, though this is 15 years ago now so might have changed since. arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When art…

This is not the case in computer science and particularly machine learning, especially in recent years. You'll find many papers where a majority of references are to preprints that stay preprints for ever. You'll also find many papers that have hundreds of citations, all while remaining preprints forever (and many of those citations are from forever-preprints themselves). In machine learning, for the most part, arxiv…

But those Arxiv papers which were not published elsewhere but which got many citations, those were read by others, i.e. reviewed by peers, i.e. peer-reviewed.

Arxiv only lacks the initial quality filter by peer review.

I'm also working in the field of machine learning. In those niche fields I work more specifically (speech recognition), I can usually still get a lot out of Arxiv-only papers. I can pretty easily see the main idea and see if there is some usefulness in the paper or not w.r.t. my own research e.g. by good experimental analysis. In don't really feel overwhelmed in the amount of papers. I don't really see the problem.

Re: Arxiv.org reaches a milestone and a reckoning

#74

Post by politician »

TLDR: 6% of submissions receive a hold, 2% are rejected. The problem, the article says, is that only 13% of the moderators are women and there is not enough diversity of nationality.

My opinion here, but it sounds like instead of a 92% immediate acceptance rate, the author would prefer an immediate acceptance rate closer to 100%. I’m not sure why diversity would get them there, but I am encouraged that the current moderators are able to produce 92% immediate acceptance rate. On the other hand, I am distraught by the suggestion that a highly diverse moderator base would presumably only account for a few percent reduction in holds and rejections. It’s my understanding that that should have a far more significant impact.

Re: Arxiv.org reaches a milestone and a reckoning

#75

Post by lynndotpy »

_def wrote: post #28

I'm in no way involved in academic research, and I don't now how it works in detail, but a question that popped into my head these days was: is there a place where people can publish papers even without any association with anything? Like, if I manage to do proper research, even without any qualifications on paper, would it still be useful? Does this make any sense?

As others have said, you can post to academic conferences. But each field has different peculiarities and patterns. You would start by doing your research, finding a relevant conference, and writing a paper to match their style.

A more approachable avenue could be blogposts and videos. One good example of a scientific blogpost I recall is here: ( https://dynomight.net/2020/12/15/some-real-data-on-a-DIY-box... ) Someone tested very simple air purifiers with limited experiments. It follows the same "abstract-methods-results-discussion-conclusion" format you might expect. Some obvious caveats, this data could have all been faked, and it isn't peer reviewed for methodological flaws. But you can reproduce the experiments yourself for ~$150.

I am in academia but I like blogposts a lot more. I wish they were more popular. I think HTML makes more sense than PDF, and to some degree they cut the middleman out between communicating to peers and communicating to other academics.

Re: Arxiv.org reaches a milestone and a reckoning

#76

Post by UncleMeat »

marginalia_nu wrote: post #71
UncleMeat wrote: post #67

Earlier quoted context omitted.

I think the core problem here is the layperson perception of peer review. People often say "it is a peer-reviewed paper!" when trying to claim that a paper is definitive and powerful. But when you speak to scientists they have a much more muted understanding of peer review. And I think that is okay. The amount of effort it would take to do a very thorough analysis of a paper is large, sometimes even approaching the c…

> In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way. This is actually deeply problematic, as a part of what's driving the replication problems is specifically publication bias. If the only thing that gets published are unexpected or noteworthy results, you're selecting for…

I don't really agree. I think it is valuable to push novel and influential results into visible venues and to reward work of this kind. It is critical to understand how this introduces bias into the process and to consider this when reading a given paper, but I'm not sure that a review system that only considers experimental rigor would be desirable.

Re: Arxiv.org reaches a milestone and a reckoning

#77

Post by raphlinus »

throwaway81523 wrote: post #45

I wonder if anyone remembers Advogato and its certification/reputation system. It claimed to be very difficult to subvert. I'm sure that was naive and it held up because nobody back then was that motivated to game it, but even today, a little human oversight could probably steady it. So I wonder if something like it could be used for arxiv, relieving some of the endorser/moderation stuff that they have now.

Yes, I bet you could do something interesting there, but one of the lessons is that social graphs can become very noisy, especially if people have an incentive to add edges. Don't expect a purely technical (reputation) system to solve what is a social/cultural problem.

Re: Arxiv.org reaches a milestone and a reckoning

#78

Post by Cupertino95014 »

YeGoblynQueenne wrote: post #60
marginalia_nu wrote: post #46

Earlier quoted context omitted.

> Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?

I think this is like the difference between homeopathy and medicine. If you're sick and go to a doctor, you have a chance that the doctor will find a way to cure you. If you're sick and go to a homeopath, there's no chance that the homeopath will find a way to cure you. Similarly, peer review has a chance to catch errors before they make it to print, while absence of peer review has no such chance. Peer review is not…

> If you're sick and go to a homeopath, there's no chance that the homeopath will find a way to cure you

Wrong, sorry. Disclaimer: I'm not a homeopath and don't go to any. Most of them are, in fact, quacks.

There are, unfortunately, chronic diseases which can be treated but not cured. Mainstream doctors and Big Pharma make a living off of those. If you have one of those, "well, why NOT try a homeopath?" is a perfectly rationale response.

"Sinus rinsing" is something that might be termed "homeopathy." It IS medically respectable, unlike most of their "treatments" (like magnets).

Re: Arxiv.org reaches a milestone and a reckoning

#79

Post by bee_rider »

krastanov wrote: post #72
monkeyjoe wrote: post #57

Earlier quoted context omitted.

I think it's because the publisher retains copyright. There is a limit on how "done" the manuscript can be and still be shared for free online. Some universities have started to fight back against this by limiting the scope of copyright restrictions that publishers can impose.

No, in most of physics there is no such limit in practice. The only difference between my published work and the preprint arxiv versions is the font and whether the layout is two columns or one column. They are word-for-word identical with identical figures.

Is it possible that you broke the rules of your journal, but nobody mothers going after a single researcher?

Re: Arxiv.org reaches a milestone and a reckoning

#80

Post by simplestats »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

> I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it.

The situation for conferences in CS is basically how it works in journals in other fields. There may be more low-quality submissions to journals, but at the same time, the extreme competitiveness means most high-quality submissions are rejected also. The "solution" has been the appearance of open-access journals which are less concerned with subjective reasons for rejection like "impact", and scale to accept more papers. But they cost thousands of dollars. Open archives are a more democratic solution.

Post reply on HN