It would be interesting to see the server getting hacked with all the papers in the open, and then the researchers sending a sorry email to the publishers.
amen
A plan to mine the world’s research papers
11–20 of 35 posts
Re: A plan to mine the world’s research papers
#12> No one will be allowed to read or download work from the repository, because that would breach publishers’ copyright. Instead, Malamud envisages, researchers could crawl over its text and data with computer software, scanning through the world’s scientific literature to pull out insights without actually reading the text.
I find this totally unconvincing. The average scientific article isn't any good, and the NLP algorithms that do tasks like this are even worse.
For scientific literature to be useful to those of us outside the academy, we need to be able to see the full document - what methodology was employed, what assumptions were made, how the data was gathered - just to be able to gauge whether the authors had any idea what they were doing. Ideally we would also be able to search over documents, explore the citation tree in some sort of UI, and access articles in multiple formats (sometimes you want a PDF, sometimes you want plaintext, and I imagine that the mathy-types might like LaTeX source).
I applaud Malamud's efforts to overcome this problem, and I hope his results prove me wrong. I just think it's sad that we have to resort to hacks like this to overcome what is obviously enormous scam that is stealing our tax dollars and stifling academic and economic creativity.
Re: A plan to mine the world’s research papers
#13Earlier quoted context omitted.
But storage costs have dropped by orders of magnitude since then and 'personal copies' of SciHub are a definite possibility.
I've read somewhere a couple of years ago that SciHub was about 35TB of data, quite big but still manageable.
$800, considering $200 for one 10TB HDD.
Re: A plan to mine the world’s research papers
#14It would be interesting to see the server getting hacked with all the papers in the open, and then the researchers sending a sorry email to the publishers.
Right, "hacked". More likely, a public-interest leak. Because, as Facebook (for example) is finding, leaks happen.
Re: A plan to mine the world’s research papers
#15So, sci-hub, but less accessible?
Also, marginally less likely to suddenly vanish in a poof of lawsuit one day. See Napster.
Well, there are torrents available, if you have the storage.
I'm confident that there are many backups. And the data could be hosted in takedown-resistant ways. It arguably is already, in Russia. Given the current political situation, I doubt that Putin would sell them out.
Re: A plan to mine the world’s research papers
#16> No one will be allowed to read or download work from the repository, because that would breach publishers’ copyright. Instead, Malamud envisages, researchers could crawl over its text and data with computer software, scanning through the world’s scientific literature to pull out insights without actually reading the text. I find this totally unconvincing. The average scientific article isn't any good, and the NLP a…
Yeah, this is silly. Elsevier and the like should be burned down for crimes against humanity. I mean, it generally is illegal to price gouge during famines.
Re: A plan to mine the world’s research papers
#17> No one will be allowed to read or download work from the repository, because that would breach publishers’ copyright. Instead, Malamud envisages, researchers could crawl over its text and data with computer software, scanning through the world’s scientific literature to pull out insights without actually reading the text. I find this totally unconvincing. The average scientific article isn't any good, and the NLP a…
Yes. I worked with a company in India looking to mine biology papers with information on genes and automatically populate a database from that. Given the various ways in which people wrote (required to pass anti-plagiarism checks as well as different writing styles) it turned out that any kind of automated annotation was rife with errors. Given that it was supposed to be used for developing drugs they dropped it in favor of hiring Master students part time at $200/month and annotating manually.
Re: A plan to mine the world’s research papers
#18Earlier quoted context omitted.
But storage costs have dropped by orders of magnitude since then and 'personal copies' of SciHub are a definite possibility.
I've read somewhere a couple of years ago that SciHub was about 35TB of data, quite big but still manageable.
It's a bit more now, going on 80T or so. Still quite manageable. A large NAS will do.
Re: A plan to mine the world’s research papers
#19R.I.P. Aaron Swartz https://en.wikipedia.org/wiki/Aaron_Swartz
Re: A plan to mine the world’s research papers
#20Earlier quoted context omitted.
I've read somewhere a couple of years ago that SciHub was about 35TB of data, quite big but still manageable.
It's a bit more now, going on 80T or so. Still quite manageable. A large NAS will do.
I just happen to have a 80T NAS that I plan to use as a sci-hub mirror, is there a way to download it? libgen torrents are dead.