The business of extracting knowledge from academic publications

The legal costs make this a non-starter unless it's done by a giant company. Who would, in my opinion, ruin it, and the odds of enough academics complying with a big tech company are small imo.

It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them.

Building this would be my dream job, but hahaha no.

The article's core claims are:

> Extracting, structuring or synthesizing "insights" from academic publications (papers) or building knowledge bases from a domain corpus of literature has negligible value in industry.

> Most knowledge necessary to make scientific progress is not online and not encoded.

> Close to nothing of what makes science actually work is published as text on the web

> The tech is not there to make fact checking work reliably, even in constrained domains.

> Accurately and programmatically transforming an entire piece of literature into a computer-interpretable, complete and actionable knowledge artifact remains a pipe dream.

It also states existing old school "biomedical knowledge bases, databases, ontologies that are updated regularly", with Expert Entry cutting through the noise in a way that NLP cannot.

Although I disagree with its conclusions, much of this jives with my experience. From the perspective of research, modern NLP and transformers are appropriately hyped but from the perspective of real world application, they are over-hyped. Transformers have deeper understanding than anything prior, they can figure out patterns in their context with a flexibility that goes way beyond regurgitation.

They are also prone to hallucinating text, quoting misleading snippets, require lots of resources for inference and enjoy being confidently wrong at a rate that makes industrial use nearly unworkable. They're powerful but you should think hard about whether you actually need them. Most of the time their true advantage is not leveraged.

-----

My disagreements are with its advice.

> For recommendations, the suggestion is "follow the best institutions and ~50 top individuals".

But this just creates a rich get richer effect and retards science since most are reluctant to go against those with a lot of clout.

> Why purchase access to a 3rd party AI reading engine...when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary). Would you invest in automation if you have billions of disposable income and access to cheap labor? After talking with employees of huge companies like GSK, AZ and Medscape the answer is a clear no.

This reminds me of responses to questions of the sort: "Why didnt't X (where X might be Ottomans or Chinese) get to the industrial revolution first?".

Article also warns against working on ideas such as "...semantic search, interoperable protocols and structured data, serendipitous discovery apps, knowledge organization."

A lot such apps are solutions chasing after a problem but could work if designed to solve a specific real world problem. On the other hand, an outsider trying to start a generalized VC backed business targeting industry is bound to fail. In fact, this seems a major sticking point in the author's endeavor.

Industry is jaded and set in their ways, startups focus on summarization and recommendations and retrieval which are low value in scientific enterprise and academia is focused on automation which turns out brittle. Still, this line of research is needed. Knowledge production is growing rapidly while humans are not getting any smarter. Specialization has meant increases in redundant information, loss of context and a stall in theory production (hence "much less logic and deduction happening").

While the published literature is sorely lacking, humans can with effort extract and or triangulate value from it. Tooling needs to augment that process.

I agree.

I'm an academic librarian, and they're completely different ways of working: When I do academic work, I (ideally) have to take my time and I'm not supposed to present my work until it's developed enough that I'm confident it presents a substantial improvement; I have to prove that it's worth a colleague's time to engage with by meeting certain requirements. Coding/developing, on the other hand, requires a lot more back and forth, a lot more "I don't know", and is more immediate in a way I find very satisfying.

I would LOVE to see more back and forth between engineers and academics in terms of ways of working; I think there's a lot of benefit to be gained there: Tech tends to not consider the future as much as they should, but the academics could really benefit from doing what you mentioned and improve the system they work in rather than accepting it.

One of the things I'm trying to do is get better at/learn some ML so I can play around with turning the things I learned in grad school into useful tools, but I'm a single journeyman dev doing this in my spare time so odds of anything actually useful coming out of it is small.

As an academic, all my papers from the last 5 or so years have associated github repos where all the code is accessible under free licenses. Most of my peers in academia do the same. Documentation quality is admittedly quite hit-and-miss, because we aren't paid for that and we need to jump to the next paper, but all the code is there and everything can be replicated even if it takes some effort due to rushed code or suboptimal documentation.

Industry is a different world, and indeed there are plenty of opaque industry papers that aren't replicable at all because much of the model is essentially a trade secret, and the paper is more an avenue for bragging than for developing new knowledge together with the rest of the community. To be honest, I would just outright disallow that kind of papers. But that's not a popular opinion, and taking into account that big tech companies sponsor our conferences and provide grants, I can't blame those who think otherwise.

AlphaZero cannot chat and interact outside moving pieces. For science, self-training would be too wasteful, intractable and impossible to boot, given there's no simulator.

An AlphaZero for science would instead be like the recent deepmind paper where the pattern matching capabilities and internal features of a neural network were used to navigate some domain's decision space of conjecture formation and testing.