Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arxiv.org reaches a milestone and a reckoning

scientificamerican.com

Post reply on HN
Print view View original

31–40 of 91 posts

Re: Arxiv.org reaches a milestone and a reckoning

#31

Post by YeGoblynQueenne »

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

So what, you'll say, independent experts can catch your mistake if they read the paper on arxiv and contact you to suggest a correction. Yes, but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it (who didn't do their due dilligence). Then errors start cascading. Science is self-correcting, sure, but it's much easier to avoid errors in the first place. That's an important function of modern peer review.

I'm speaking from experience, of course. Peer review has certainly helped improve my work. I didn't enjoy having to redo the work, but the end result was something I could be confident about. Also, I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it. You might even see your article posted on HN. When was the last time a paper published on a conference server was posted on HN? See?

Re: Arxiv.org reaches a milestone and a reckoning

#32

Post by EGreg »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

Well, this is exactly why we should have more WikiNews and less Twitter. I explore these issues in far more depth here:

https://www.youtube.com/watch?v=gv5mI6ClPGc

(Discussion w Noam Chomsky)

Re: Arxiv.org reaches a milestone and a reckoning

#33

Post by IshKebab »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors."

I agree with the rest of your points though. Zero peer review is probably not a great solution.

Re: Arxiv.org reaches a milestone and a reckoning

#34

Post by astrange »

WHA8m wrote: post #20
ahelwer wrote: post #9

Earlier quoted context omitted.

Because you cannot rely on the rich to fund common goods through voluntary donations. It is not a good idea and never will be. Other than some medical research, charity has never solved a single social ill.

Discard the last sentence. You're talking in absolute terms and that's leading nowhere. It's been some time, but this is what came to my mind immediately: https://en.wikipedia.org/wiki/Fuggerei That you can not rely on the rich to put their money in good use with the general public in mind is right. Does it happen? Yes. Does it happen often? Definitely not.

The anti-charity people get around their absolutism by inventing "mutual aid", which is what they call charity when they like it.

(similar thing in education involved deciding "equality" was bad, so they invented "equity", which is the same thing except they like it.)

Re: Arxiv.org reaches a milestone and a reckoning

#35

Post by mam2 »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

No post body was provided.

Re: Arxiv.org reaches a milestone and a reckoning

#36

Post by SiempreViernes »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

There's actually some incentive: you get to feel smart by telling someone else is wrong. But it is indeed not a very strong one, and there's plenty of other places to act smart on the internet where you'll actually have an audience too!

Re: Arxiv.org reaches a milestone and a reckoning

#37

Post by marginalia_nu »

IshKebab wrote: post #33
YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors." I agree with the rest of your points though. Zero peer review is probably not a great solution.

Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate.

It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).

Re: Arxiv.org reaches a milestone and a reckoning

#38

Post by lN5uhiNxWPBi »

adamnemecek wrote: post #2

I legit don’t get how come there isn’t some government funded publication platform. No other endeavor has as much bang for buck. Arxiv needs $2.5m a year https://arxiv.org/about/reports-financials which is chump change.

There is, right? In many cases around the world.

Here in the UK, there is a big research evaluation exercise called the REF. The REF is important because it largely determines how the government allocates funds. In order to be eligible to be evaluated, researchers have to essentially submit their pre-prints into open access.

What this has meant for the last 10+ years is that most universities participating in the REF have an e-prints server, like the arxiv. Obviously, being publically funded, universities are then using government money to pay for these costs.

Now your question might be whether it's sensible for the entire country (or research councils) to operate their own e-prints server. I don't know the answer to this.

All I know is that the more bureaucracy you involve, the more bullshit results.

Re: Arxiv.org reaches a milestone and a reckoning

#39

Post by hasjekyll »

YeGoblynQueenne wrote: post #31

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll sa…

Agreed. Peer-review is essential. I do like the idea of a public peer review however, similar to that in the https://joss.theoj.org/ - I had a really pleasant experience reviewing a paper for JOSS.

I think sometimes reviewers hide behind their anonymity to give snarky responses, and using their power over you, knowing you will try your very best to implement all of their suggestions in order to publish faster. I just think the whole process should be more transparent.

Re: Arxiv.org reaches a milestone and a reckoning

#40

Post by hasjekyll »

marginalia_nu wrote: post #37
IshKebab wrote: post #33

Earlier quoted context omitted.

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors." I agree with the rest of your points though. Zero peer review is probably not a great solution.

Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate. It's very po…

Completely agree - the quality of peer-review is a lottery. Sometimes you get an excellent referee making good points, other times you get a referee solely focussed on making sure you cite their work. And, occasionally, you get a referee who doesn't even appear to read the paper and just waves it through.
Post reply on HN