Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The business of extracting knowledge from academic publications

markusstrasser.org

Post reply on HN
Print view View original

21–30 of 122 posts

Re: The business of extracting knowledge from academic publications

#21

Post by bluGill »

jokteur wrote: post #9
hoseja wrote: post #8

Earlier quoted context omitted.

As the article states, papers are mostly career advancement tools and scientists are incentivized to put the least amount of useful information into them they can get away with. Real scientists mostly learn from their instructors who possess all the jealously guarded institutional knowledge. Yes, it is very broken.

It is always difficult to try to understand and implement the theory explained in papers which seem fine on the surface, but when you look more closely, you find a bunch of mistakes, there are giant holes in the details, and you end up trying to redo the whole paper. There should be journals/websites/blogs dedicated to trying to reexplain / implement papers.

Doing a good review is hard.

I've been reviewing a few C++ papers (things proposed to C++23) lately. Many of them are over my head and all I can find are a few spelling errors. The ones I've understood took me 3 readings before I found some giant holes in the details (which I pointed out to the authors, the next revision corrected them). In one case I actually started implementing the feature myself, and only then did I realize there was something ambiguous in the paper (after talking with the author we decided to document an implementors choices as it doesn't matter for the 99% use case, and the rest could go either way depending on hardware and data details so better to allow choice)

The vast majority if papers I'm far too laze to go into that level of detail on. I just assume the authors is a smart person and so I trustingly let them go. It may well be if I understood the paper I'd be horrified, ask me in 2043 when we have 20 years of hindsight...

I have to believe that peer review is the same - many reviewers are just reading and looking for something obvious but not really understanding details.

Re: The business of extracting knowledge from academic publications

#22

Post by anonymousDan »

hoseja wrote: post #8
civilized wrote: post #4

Interesting but I don't know how to make sense of it. How can it be that "close to nothing of what makes science actually work is published as text on the web"? - Is the information that makes science actually work mostly in images that the machines don't yet understand? - Was the information paywalled or in private databases and inaccessible to this researcher? - Are the papers mostly just advertisements for researc…

As the article states, papers are mostly career advancement tools and scientists are incentivized to put the least amount of useful information into them they can get away with. Real scientists mostly learn from their instructors who possess all the jealously guarded institutional knowledge. Yes, it is very broken.

I mean honestly this is just total bullshit. There is plenty of value in academic papers. It's just that there is very little money to be made in developing tools such as those mentioned by the OP as there is very little money in academia.

Re: The business of extracting knowledge from academic publications

#23

Post by urthor »

civilized wrote: post #6
bryanph_ wrote: post #3

One thing that strikes me about most academic knowledge tools is that they seem to focus on parsing the current set of academic literature and producing supposedly interesting insights out of them (which quickly tends to snowball into wanting some kind of generalized model for knowledge as a whole). What I think is much more interesting is creating tools that help people create better academic writing in the first pl…

To your point but even more general: the ML/AI space is far too focused on replacing people rather than helping people. There is a suffocating cultural conceit that we are on the verge of general AI and oh my gosh what will the humans do, we better institute universal basic income right away, etc. What a joke. Try to help humans think better first. If you succeed at that, you might be on the right track towards devel…

Unfortunately you'll run into the rapid fact that in the ML/AI space you get almost zero points for building something.

You get a whole lot of points for discovering something, designing something, or a proof.

But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better.

Mostly because everyone quietly understands all the extraordinarily complicated mathematics is actually extraordinarily complicated.

Hence the ROI isn't worthwhile.

Re: The business of extracting knowledge from academic publications

#24

Post by Kydlaw »

Interesting insights. Particularly on the business aspect. But I am not surprise by the outcome, as the author said: nobody want to pay for what its proposed in academia. Everybody is already more or less struggling with funding, so nobody want to add extra fat in their funding requests.

Coming from CS, something I would really like to see though is a tool that would summarize a scientific area/domain. Something that would kill literature reviews and/or would provide an overview of the hot topics/open questions in different areas.

Edit: corrections

Re: The business of extracting knowledge from academic publications

#25

Post by jakub_g »

cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive:

In opensource the attitude is "See bug? Send a PR!"

Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.

Re: The business of extracting knowledge from academic publications

#26

Post by PaulHoule »

When you count all the ways to be wrong, the median scientific paper is wrong.

In biomedical fields they dismiss more than half of papers out of hand when they do a Cochrane meta analysis. It begs the question of why such papers (which aren’t fit to extract knowledge from) are published or funded at all.

I got a PhD in theoretical physics and routinely found that something was wrong on page 23 of a 50 page calculation and nobody published anything about it in 30 years. Possibly the whole body of work on string theory since 1980 [most of hep-th] is a pipe dream at best. Because young physicists have to spend the first third of their career in a squid game fight for survival not to fathom the secrets of the universe but to please their elders we get situations like that absurd idea of Steve Hawking that information gets lost in a black hole. (E.g. if you believe that you aren’t even going to try quantum gravity.)

Re: The business of extracting knowledge from academic publications

#27

Post by remram »

jakub_g wrote: post #25
cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive: In opensource the attitude is "See bug? Send a PR!" Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact…

The biggest difference IMHO is when comparing to something like Wikipedia or Stackoverflow. I wish the fabric of scholarly communication similarly allowed for browsing reviews, updating papers, commenting with new references, etc.
Post reply on HN