Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arxiv.org reaches a milestone and a reckoning

scientificamerican.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: Arxiv.org reaches a milestone and a reckoning

#61

Post by thereddaikon »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

Peer review is important but lets be sure to not conflate it with the established way it operates. There is nothing that says peer review can only happen through submitting to an established Journal that conducts a traditional peer review process. We already know that process isn't perfect so there is no harm in looking to improve or replace that. Does that mean ArXiv as it is works as a replacement? No, and you make some good points why it doesn't.

Re: Arxiv.org reaches a milestone and a reckoning

#62

Post by YeGoblynQueenne »

atomack wrote: post #44
YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

The way we used arxiv worked well in physics, though this is 15 years ago now so might have changed since. arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When art…

This is not the case in computer science and particularly machine learning, especially in recent years. You'll find many papers where a majority of references are to preprints that stay preprints for ever. You'll also find many papers that have hundreds of citations, all while remaining preprints forever (and many of those citations are from forever-preprints themselves).

In machine learning, for the most part, arxiv is used to avoid peer-review. Or a way to "publish" work that has been rejected by a peer-reviewed publication, of course.

And to be more cynical, it's also a convenient source of references to pad up a Related Work section and make it look like incremental work is part of a growing body of groundbreaking new work. /jaded

Edit: well, I'm not just being cynical. The fact that everyone can put their half-baked papers on arxiv means that the 90% of work that is crap, per Sturgeon's Law, is now a much bigger quantity than ever before and one must sift through reams and reams of crap before finding work that has any meaningful results to report. Again, that's the case in machine learning specifically. I don't know about other fields.

Re: Arxiv.org reaches a milestone and a reckoning

#63

Post by erwincoumans »

IAmEveryone wrote: post #6
adamnemecek wrote: post #2

I legit don’t get how come there isn’t some government funded publication platform. No other endeavor has as much bang for buck. Arxiv needs $2.5m a year https://arxiv.org/about/reports-financials which is chump change.

If you mean some government should fund arXiv, specifically, then I don't disagree but would guess the government's default position of "it exists without public money, and we think that's rather excellent". If you're suggesting it as alternative for academic publishing in general, it would cost a lot more. Arxiv isn't a "publication platform" in the sense that peer-reviewed journals are. The cost of hosting and dist…

Interestingly, the peers doing the hard work (of actual reviewing) for free.

Running the reviewing process can also happen through https://openreview.net

Not sure why we need Elsevier exactly, could you be more specific?

Re: Arxiv.org reaches a milestone and a reckoning

#64

Post by leephillips »

ahelwer wrote: post #9
thrwn_frthr_awy wrote: post #3

Earlier quoted context omitted.

I also don't understand how there can so much money to throw at startups and not one person to "adpot-a-highway" for a year.

Because you cannot rely on the rich to fund common goods through voluntary donations. It is not a good idea and never will be. Other than some medical research, charity has never solved a single social ill.

Has any social ill been eliminated by any other means?

Re: Arxiv.org reaches a milestone and a reckoning

#65

Post by YeGoblynQueenne »

HerrMonnezza wrote: post #50
YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

> What you lose on arxiv is peer review. Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-) You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing. In other words: I think that an article on arXiv (…

Well, if all one wants is an "audience" one can simply report the results of their research on twitter. Much bigger audience. But that's not the point, is it? The point is that you want someone who knows their shit, to review your paper, and who will have both the knowledge and the motivation to engage with your work critically and help you see what is behind all those big blind spots we all have for our own work. I'm sure this happens with papers put on arxiv, occasionally, but my expectation is that just because the paper is on arxiv most people will not be that interested in making a serious effort to review it, and that it is much more likely that such a serious effort will be made by a reviewer in a peer-reviewd venue, particularly one of the top journals (conference reviews are a mess).

So, yeah, I disagree. The level of scrutiny one gets from putting their work on arxiv doesn't compare with peer review by experts in one's field.

Re: Arxiv.org reaches a milestone and a reckoning

#66

Post by time_to_smile »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

This is a remarkably naive (and charmingly optimistic) view of peer-review.

For the vast majority of the history of academic research peer review did not play a major part. It wasn't until the 1970s that the modern day peer review system emerged and it did so as a reaction to a funding crisis in academia to produce the illusion of legitimacy.

It of course has done no such thing. It has done nothing to limit the reproducibility crisis which has shown it's ugly head in nearly all scientific fields of study. One could even argue that peer review coupled with a publish or perish culture is the cause of such a failure of science.

This also means just about any great scientific discovery you can think of prior to 1970 did not go through the peer review process as we know it today. If science seems less exciting today, peer review is certainly one of the reasons for this. I'll leave you with Geoffrey Hinton's words on the subject:

> Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Laking peering review is a feature, and not just because it speeds up time to results.

Re: Arxiv.org reaches a milestone and a reckoning

#67

Post by UncleMeat »

marginalia_nu wrote: post #46
Karellen wrote: post #41

Earlier quoted context omitted.

> in several fields, 50-90% of peer-reviewed studies fail to replicate. Is that a failure of peer-review though? Or is it more of a consequence that even if you do everything right you'll still get some false positives some of the time, and negative results tend not to get published at all? (i.e. the problem that pre-registration for medical trials hopes to solve - https://en.wikipedia.org/wiki/Preregistration_(scien…

> Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?

I think the core problem here is the layperson perception of peer review. People often say "it is a peer-reviewed paper!" when trying to claim that a paper is definitive and powerful. But when you speak to scientists they have a much more muted understanding of peer review. And I think that is okay. The amount of effort it would take to do a very thorough analysis of a paper is large, sometimes even approaching the cost to do the research in the first place.

In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way.

A few fields (psych is the big one) are funding replication grants. That'd be the true mark of quality.

Re: Arxiv.org reaches a milestone and a reckoning

#68

Post by bopbeepboop »

YeGoblynQueenne wrote: post #65
HerrMonnezza wrote: post #50

Earlier quoted context omitted.

> What you lose on arxiv is peer review. Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-) You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing. In other words: I think that an article on arXiv (…

Well, if all one wants is an "audience" one can simply report the results of their research on twitter. Much bigger audience. But that's not the point, is it? The point is that you want someone who knows their shit, to review your paper, and who will have both the knowledge and the motivation to engage with your work critically and help you see what is behind all those big blind spots we all have for our own work. I'…

In mathematics, it’s a continuum:

Discussion starts in blog posts, on Twitter, in conversations.

Then longer blog posts and code demos.

Then pre-prints and more blog posts about that.

Then finally journal.

You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong.

Re: Arxiv.org reaches a milestone and a reckoning

#69

Post by telxosser »

buro9 wrote: post #19
henrydark wrote: post #5

The two sites I start my day with: hacker news and arxiv.

Do you read the /new submissions for particular topics? HN is a nice tidy list of top things, is there an equivalent for ArXiv?

arXiv advanced search is great.

I just have a few bookmarked searches for what I am interested in.

I don't see how you could generalize things though across all papers. Like randomly clicking on what was submitted to High Energy Physics - Lattice yesterday. I don't have the first clue what any of that is and that would be the same for most topics for most people I imagine.

Re: Arxiv.org reaches a milestone and a reckoning

#70

Post by telxosser »

mananaysiempre wrote: post #13
ahelwer wrote: post #9

Earlier quoted context omitted.

Because you cannot rely on the rich to fund common goods through voluntary donations. It is not a good idea and never will be. Other than some medical research, charity has never solved a single social ill.

The introductory econ course I once took mentioned the (apparently classic) lighthouse study[1], though now that I did a web search it doesn’t seem that unequivocal. Many 19th century projects were funded through public participation (the Eiffel Tower and the Statue of Liberty come to mind), even if the social value of some of them was questionable (though I think some railroads were built this way too?). Before that…

You can just visit Pittsburgh to see how preposterous this is. The city is practically unimaginable culturally without Andrew Carnegie's wealth.
Post reply on HN