Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The business of extracting knowledge from academic publications

markusstrasser.org

Post reply on HN
Print view View original

61–70 of 122 posts

Re: The business of extracting knowledge from academic publications

#61

Post by Mezzie »

anyfactor wrote: post #14

> Why purchase access to a 3rd party AI reading engine or a knowledge graph when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary) I really like jobs people think AI can do in theory but can't really do them effectively irl. Where do I get a part-time gig like that if I think I am capable of reviewing and creating summary of non-STEM papers? Except for homework…

Yeah, you can't outsource that to Hyderabad. You'd need to know subject knowledge + very specific English and possibly other languages depending on the field (not saying Indians can't do this, but I've studied enough languages to know that doing high level/academic work in a non-native language is hell even when the language is pitched to students).

And you'd have to know enough about the process and authors to know what makes papers relevant. The metadata matters as much as the data.

Re: The business of extracting knowledge from academic publications

#62

Post by Mezzie »

bluGill wrote: post #21
jokteur wrote: post #9

Earlier quoted context omitted.

It is always difficult to try to understand and implement the theory explained in papers which seem fine on the surface, but when you look more closely, you find a bunch of mistakes, there are giant holes in the details, and you end up trying to redo the whole paper. There should be journals/websites/blogs dedicated to trying to reexplain / implement papers.

Doing a good review is hard. I've been reviewing a few C++ papers (things proposed to C++23) lately. Many of them are over my head and all I can find are a few spelling errors. The ones I've understood took me 3 readings before I found some giant holes in the details (which I pointed out to the authors, the next revision corrected them). In one case I actually started implementing the feature myself, and only then di…

SO difficult, especially given that just because you're in the 'same' field and technically qualified to do a peer review doesn't mean you actually understand what you're reading.

For example, I'm qualified to review papers on educational programs for children. I should never be asked to do that.

Re: The business of extracting knowledge from academic publications

#63

Post by Mezzie »

TOMDM wrote: post #32
eurasiantiger wrote: post #7

Earlier quoted context omitted.

It likely wouldn’t take much to craft an ”arXiv Copilot” out of GitHub Copilot.

"It's well understood how to" And "It likely wouldn't take much to" Are worlds apart in this case, training and deploying models on that scale is a huge investment, even if you already had all the code and cleaned training data.

Can confirm: This is my main tech interest at the moment and if I consider how long it's going to take, I want to die.

Re: The business of extracting knowledge from academic publications

#64

Post by captainmuon »

cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

I may be a bit cynical, but at least in my former field (experimental physics), the main purpose of papers seems to be to "lock in" a finished achivement. You do the actual research, pass internal reviews and peer review, and then publishing the paper is just to make it "official". Unfortunately, many papers are never expected to be read. The crucial information exists, but you usually get it from personal communication, internal wikis, or review articles. You just need the paper to copy a formula or graph, and to cite it in the end.

There are papers that are well-written and useful, but there are at least as many that are just drivel (I probably contributed to both kinds).

Unfortunately, the prevailing attitude is that outside people will not understand our stuff anyway, so we often make no effort to make papers understandable, or to publish data. (There is a lot of great outreach and science communication, but not so much for students or researchers from other fields who want to follow the technical details.)

Re: The business of extracting knowledge from academic publications

#65

Post by auggierose »

geoduck14 wrote: post #44
urthor wrote: post #23

Earlier quoted context omitted.

Unfortunately you'll run into the rapid fact that in the ML/AI space you get almost zero points for building something. You get a whole lot of points for discovering something, designing something, or a proof. But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. Mostly because everyone quietly understands all the extraordinar…

>But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. I can't speak to each and every person working on ML, but I thought I would share a fun use case I ran across the other day. There is a business in some foreign country that is similar to Uber Eats: customer goes to an app, browses for food from various restaurants, orders…

Sounds like this would be an epic fail. I mean, just add a spoon of oil more, and your calorie guess is totally off. This is a clear cut case of SEEMINGLY helping. It certainly does not help the end customer. It might help the restaurant, as they don't care if the customer is receiving valid information, as long as they are buying.

Re: The business of extracting knowledge from academic publications

#66

Post by mellavora »

bluGill wrote: post #21
jokteur wrote: post #9

Earlier quoted context omitted.

It is always difficult to try to understand and implement the theory explained in papers which seem fine on the surface, but when you look more closely, you find a bunch of mistakes, there are giant holes in the details, and you end up trying to redo the whole paper. There should be journals/websites/blogs dedicated to trying to reexplain / implement papers.

Doing a good review is hard. I've been reviewing a few C++ papers (things proposed to C++23) lately. Many of them are over my head and all I can find are a few spelling errors. The ones I've understood took me 3 readings before I found some giant holes in the details (which I pointed out to the authors, the next revision corrected them). In one case I actually started implementing the feature myself, and only then di…

Someone once said that they enjoyed one of my papers, and that even though they thought the writing was very clear they still had to read it 3 times to understand it.

I told them that I had to write it 100 times and spend two years before I understood it.

So if they could pick it up in 3 readings over 3 days, they were doing pretty good.

Re: The business of extracting knowledge from academic publications

#67

Post by tchalla »

cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

> I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding.

If you think the entire field of academia doesn't achieve any purpose, you may want to reconsider your position. Most likely, almost everything that you do today on a computer was an academic paper. Yes, it was without code and data. Yet, it was not unusable and achieved more than enough purpose.

The average comment on HN on academia comes from a mindset where everyone wants a product. The purpose of a paper is NOT to release a software or a product. But, to test an idea under some assumptions. That's what all research does at its core - formulate a hypothesis, design an experiment to test the hypothesis and report the results and implications. Are all research papers perfect"? No. Are all of them usable? No.

Your use case - sound synthesis for a specific instrument - may not be a scientific challenge. It is however an engineering challenge and hence, you found a better answer amongst hobbyists and tinkerers. Now, try looking for the a vaccine for Covid - and guess where you'd find that answer? In decades of research on mRNA with repeated failures, papers that couldn't be replicated, unavailability of "code" and samples with verbal descriptions skipping crucial details.

Re: The business of extracting knowledge from academic publications

#68

Post by temporaryi3 »

captainmuon wrote: post #64
cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

I may be a bit cynical, but at least in my former field (experimental physics), the main purpose of papers seems to be to "lock in" a finished achivement. You do the actual research, pass internal reviews and peer review, and then publishing the paper is just to make it "official". Unfortunately, many papers are never expected to be read. The crucial information exists, but you usually get it from personal communicat…

I did my phd in experimental physics and I have to say my realisation of this large point, that papers are little more than resume padding to lock in an achievement was a significant contributor towards destroying, and I use destroying seriously here, any faith or trust that peer review or publishing has anything at all to do with the scientific method at all.

Your results replicate, or they don't. Your calculations, equations, and models predict experiment. Or they don't.

Writing papers about it and getting the feedback of "peers" is nothing more than an old fashioned circle jerk for padding resumes, CVs, and persuading other people in that academic hierarchy that you deserve funding. It is a game that is divorced from actually learning, researching, understanding, measuring, and predicting the world.

Re: The business of extracting knowledge from academic publications

#69

Post by cyanydeez »

temporaryi3 wrote: post #68
captainmuon wrote: post #64

Earlier quoted context omitted.

I may be a bit cynical, but at least in my former field (experimental physics), the main purpose of papers seems to be to "lock in" a finished achivement. You do the actual research, pass internal reviews and peer review, and then publishing the paper is just to make it "official". Unfortunately, many papers are never expected to be read. The crucial information exists, but you usually get it from personal communicat…

I did my phd in experimental physics and I have to say my realisation of this large point, that papers are little more than resume padding to lock in an achievement was a significant contributor towards destroying, and I use destroying seriously here, any faith or trust that peer review or publishing has anything at all to do with the scientific method at all. Your results replicate, or they don't. Your calculations,…

There's probably space for a startup that properly archives the technical nature of findings.

Re: The business of extracting knowledge from academic publications

#70

Post by markusstrasser »

kkylin wrote: post #39
cousin_it wrote: post #16

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is bas…

I'm an academic (applied math) and want to respond to this: academic papers are the way they are for lots of reasons, many of which (not so good) have been mentioned on HN. There are a couple that I do not see very often however: (1) Many academics aren't aware non-academics read their papers at all: we work with other academics, go to conferences with other academics, and on the rare occasions we hear from readers,…

You hit the nail on the head. Will put some of that in the appendix of the post!
Post reply on HN