21–25 of 25 posts
None of that money goes to the two reviewers. The reviewers could do the same work for open access journals, and they would have (IMHO) no motivation not to do it for free.
Summary: Current peer review is not 100%; we need to think in terms of authors' motivations (tenure); try a hacker news model, possibly layered with badges-of-approval from editorial board. The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent pu…
I can skim an article posted on hacker news in 30 seconds and have a general idea of what the content was. I can also skim a research paper, but only if I know that the research was done well (e.g. all the proofs are correct, or the assumptions in the model made sense, or all correct variables were properly controlled in the experiment). If that stuff isn't guaranteed, the paper is useless because I don't know if I can use it for my research. And to figure out all that stuff myself would take a few days literally out of my research. Just because 10 people skimmed it, thought it sounded cool, and up-voted it does not mean the results in the paper are correct.
Other than this review aspect, I really like this product. If they can figure out a good incentivized review system, I think it will be a big hit.
It's also worth noting that academia.edu is a for-profit enterprise. The domain academia.edu was purchased prior to the strictures put in place to ensure that only qualified educational institutions could register .edu domains, and has been grandfathered ever since. Academia.edu itself launched in 2008 apparently. The author of this piece very much has a horse in this game, and as a result I very much view his writin…
I think you are getting the horse and the cart in the wrong order (to continue the horse analogies :) ). I started Academia.edu, after finishing my PhD at Oxford, because I believe that science, is too closed, and too slow, and that I think that we can build a faster, cheaper, and better system for disseminating research. You're wondering whether I'm asserting the things in the post because I started the company, but…
And I'm not asserting anything about your motivations for why you started a for-profit company. I agree with you that the publishing models we have currently are broken. At the same time, I'm not entirely happy with some of the changes that have taken place in the past 5-10 years.
When you look to the changes that digital distribution have wrought in industries like the news media over a similar time span, the changes are not all healthy ones either from the perspective of publishers or consumers. The fact that HuffPo and Gawker have risen to the top of the pile as the sorts of money making enterprises is not such a great example to set.
Moreover, I'm unconvinced that adding more eyeballs to papers is going to improve quality of review. If your assertion is that we can make scientific review faster and cheaper without a subsequent loss of quality (let's just say "more scalable"), then I'd love to hear your response to a claim i like to make. I'd assert that publishing scientific papers is more similar to the trials and travails of online media than it is to publishing source code.
The thing that makes open source software work is that software behavior (broadly speaking) is verifiable. I can run your code and we can agree on the results, or you can tell me i'm an idiot for not configuring the software appropriately. I'd assert that FOSS scalability derives from this ability. More eyeballs helps because everyone is working from/over the same artifact.
The same can't be said for most academic papers, because most academes don't even publish their raw data, let alone possess have the facilities or the will to publish verifiable algorithmic processes for arriving at their data.
I'd love to see a project which encouraged better replicability and verification of results/test data. But you haven't suggested any of those things.
P.S. absolutely, please do be the change you're looking for. But I'm not convinced that you're the change i'm looking for.
Earlier quoted context omitted.
As a researcher myself, I do not use the web as a means of content discovery. I do use google scholar, pubmed, and citeseer, but these are only useful because they restrict the search to just peer reviewed journals (and optionally patents in google scholar case). Trying to use general web search is often far too noisy. Often times I find interesting papers on page two, three, or four of google scholar. In a general w…
It's true that the papers in the indexes of Google Scholar, Pubmed, and Citeseer are peer reviewed, but what that shows me is that what is really driving the discovery process is the ranking system in those search engines: i.e. the order in which the hundreds of results for a given search query show up. One of the drawbacks of the existing peer review process is that academics don't get credit for their reviews, nor…
The immutability of published works is also crucial. It routine for writers to leave out details covering in prior works. This saves immense amounts of time on both writers consumers of scientific works. However, it also means that it is crucial that all cited works be preserved forever. If a document truly goes missing, then entire lines of work become incomplete. Papers can and are infrequently withdrawn, but as far as I know, the work is not erased, but merely marked as bad.
The way updates or corrections are made is via newer papers revisiting topics. But it remains that at every step some amount of decent due diligence is done to correct errors and not clutter up the records with incomplete versions.
As great as the web is for unstructured content, you cannot easily apply it to every area, and especially not to scientific publications. There are plenty of other examples of curated sources on the web that crucial. Map systems, curated databases of restaurants, directories of people like LinkedIn and Facebook, and even Wikipedia can be counted a curated system due to its system of editors.
Scientists have always and are still free to share data, white papers, and whatever else outside the peer review system. The main reason peer review is still here is that no suitable alternative has ever been proposed that addresses all the points that peer review does. There is a push towards open access journals to benefit the world at large though.
I understand that you are passionate about this, but I'm not convinced by your arguments.
Summary: Current peer review is not 100%; we need to think in terms of authors' motivations (tenure); try a hacker news model, possibly layered with badges-of-approval from editorial board. The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent pu…
I don't think the hacker news / stackoverflow model will work for research papers. People read hacker news articles because they want to keep up with what's going on in the tech community, even if it's not super related to their particular job. Researchers generally don't just read research papers for fun. Rather, they are working on their own research, and they are looking for relevant papers whose results they can…
Perhaps reviewers could publicly vouch for an article to give it credence. Reviewers, who are also researchers, can have scores based on their own papers. If a reviewer with a high score publicly vouches for an article, that article would receive points in favor of being correct.
The most interesting time in writing a paper is when it's under review before being accepted. The process is extremely similar to debugging an app shortly before a release. Part of my motivation for the proposed design is to upgrade this to an ongoing process -- analogous to the move from rare-release desktop software to dynamic web apps. If the author cares about their paper and receives useful comments, then the paper can iterate on that feedback.
I agree that many researchers in practice hear about papers they want to read through word of mouth. For this use case, I like to compare an online platform to stackoverflow (SO), in that you most often visit stackoverflow to solve a focused problem you have, as opposed to browsing for fun (like hacker news). You get an external problem to a page (a math paper / an SO question), you check it out, and you learn what you wanted to learn, or you contribute your knowledge or ask a question. It's a mini-community around a particular idea (a paper or a question). I see this as the main use case. The analogy with hacker news was to imply that the platform could be built around links to arXiv papers in the way HN is built around external news stories (while SO is more self-contained). Like HN, it doesn't seem necessary for an author to submit their own paper to the site.
Finally, I would say that some researchers do browse for fun. I enjoy reading several mathematicians' and computer scientists' blogs, and occasionally visit mathoverflow. This would be a secondary use of the platform, but it would be fun to have a front page of popular papers, and per-topic subpages.