A plan to mine the world’s research papers • Hacker News Board

Re: A plan to mine the world’s research papers

#31

Post by fromthestart » Fri, Jul 19, 2019, 3:38 PM UTC

montecarl wrote: post #29

dlkf wrote: post #12

> No one will be allowed to read or download work from the repository, because that would breach publishers’ copyright. Instead, Malamud envisages, researchers could crawl over its text and data with computer software, scanning through the world’s scientific literature to pull out insights without actually reading the text. I find this totally unconvincing. The average scientific article isn't any good, and the NLP a…

> The average scientific article isn't any good, and the NLP algorithms that do tasks like this are even worse. In my day job, I'm often tasked with implementing algorithms from recently published physics papers. In order to do so, I normally have to read through at least 10 related papers (both cited papers and cited by papers) in order to have a clear idea in my head about what is going on. Even then, I often have…

>In my day job, I'm often tasked with implementing algorithms from recently published physics papers. In order to do so, I normally have to read through at least 10 related papers (both cited papers and cited by papers) in order to have a clear idea in my head about what is going on. Even then, I often have to discuss what I have read with 2 or 3 other people and try out many different approaches before we finally understand what the paper meant

This post makes me feel slightly better about how often I come away from reading a paper feeling like I only have a shallow understanding of the content.

Re: A plan to mine the world’s research papers

#32

Post by jdjayded » Fri, Jul 19, 2019, 4:53 PM UTC

I'm a little late to this party, but here's a mandatory research plug:

My lab works in the area of evidence based medicine. My research focuses on Randomized Controlled Trials, and the overall goal is to automate (fully or partially) the meta-analysis of medical interventions. To that end, we collected a dataset of intervention pairs, statistical significance findings about these pairs, and a minimal rationale supporting the significance finding.

Since these annotations were collected on full text PubMed Open Access articles, we can distribute both the articles and the annotations: https://github.com/jayded/evidence-inference ; paper: https://arxiv.org/abs/1904.01606 ; website: http://evidence-inference.ebm-nlp.com/

We're working on collecting more complete annotation. We hope to facilitate meta-analysis, evidence based medicine, and long document natural language inference. We might even succeed (somewhat) since this is a very targeted effort, as opposed to something more broad.

Re: A plan to mine the world’s research papers

#33

Post by jacquesm » Fri, Jul 19, 2019, 5:49 PM UTC

IronBacon wrote: post #23

jacquesm wrote: post #18

Earlier quoted context omitted.

It's a bit more now, going on 80T or so. Still quite manageable. A large NAS will do.

So if the 35TB figure was correct a few years ago, I didn't expect now to be more than doubled. Do you happen to know if it's only SciHub or it also includes Libgen? If it's only SciHub that's a lot of papers... ^__^;

Just scihub.

Re: A plan to mine the world’s research papers

#34

Post by jacquesm » Fri, Jul 19, 2019, 5:50 PM UTC

m1el wrote: post #20

jacquesm wrote: post #18

Earlier quoted context omitted.

It's a bit more now, going on 80T or so. Still quite manageable. A large NAS will do.

I just happen to have a 80T NAS that I plan to use as a sci-hub mirror, is there a way to download it? libgen torrents are dead.

email?

Re: A plan to mine the world’s research papers

#35

Post by mirimir » Sat, Jul 20, 2019, 7:13 PM UTC

natechols wrote: post #25

mirimir wrote: post #16

Earlier quoted context omitted.

Yeah, this is silly. Elsevier and the like should be burned down for crimes against humanity. I mean, it generally is illegal to price gouge during famines.

Elsevier is a useless profiteer, but let's please remember that scientists are voluntarily submitting their articles to Elsevier journals, and the funding agencies (and universities) are doing nothing to stop them. I find it a little frustrating to read the wailing from institutions like UC (where I used to work) about subscription prices when they are part of the problem in the first place.

Sure. It's the prisoners' dilemma.

But that doesn't exonerate the prisons.