Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Classifying all of the pdfs on the internet

snats.xyz

Post reply on HN
Print view View original

111–117 of 117 posts

Re: Classifying all of the pdfs on the internet

#111

Post by a123b456c »

j_bum wrote: post #100
Loughla wrote: post #91

Earlier quoted context omitted.

How do you decide who is listed first? And does the ampersand symbolize something that the word and doesn't, or is that just citation style?

In biomedical research or tangential fields, author order generally follows these guidelines: First author(s): the individual(s) who organized and conducted the study. Typically there is only a single first author, but nowadays there are often two first authors. This is because the amount of research required to generate “high impact” publications simply can’t be done by a single person. Typically, the first author i…

Agree with this but it does not apply to all fields. Economists have a norm of alphabetizing author names unless the contributions were very unevenly distributed. That way authors can avoid squabbling over contributions.

Re: Classifying all of the pdfs on the internet

#112

Post by gnewton77 »

j_bum wrote: post #101
gnewton77 wrote: post #63

Did some similar work with similar visualizations ~2009, on ~5.7M research articles (PDFs, private corpus) from scientific publishers Elsevier, Springer: Newton, G., A. Callahan & M. Dumontier. 2009. Semantic Journal Mapping for Search Visualization in a Large Scale Article Digital Library. Second Workshop on Very Large Digital Libraries at the European Conference on Digital Libraries (ECDL) 2009. https://lekythos.li…

Nice article, thanks for sharing. I can imagine mining all of these articles was a ton of work. I’d be curious to know how quickly the computation could be done today vs. the 13 hour 2009 benchmark :) Nowadays people would be slamming those data through UMAP!

Thanks! Yes, I'm sure using modern hardware and modern techniques would improve both computation time and results.

Re: Classifying all of the pdfs on the internet

#113

Post by gnewton77 »

j_bum wrote: post #100
Loughla wrote: post #91

Earlier quoted context omitted.

How do you decide who is listed first? And does the ampersand symbolize something that the word and doesn't, or is that just citation style?

In biomedical research or tangential fields, author order generally follows these guidelines: First author(s): the individual(s) who organized and conducted the study. Typically there is only a single first author, but nowadays there are often two first authors. This is because the amount of research required to generate “high impact” publications simply can’t be done by a single person. Typically, the first author i…

This is a fairly accurate account of the roles played for this paper. I came up with the original idea, wrote all the code, did the analysis, and wrote the paper, with my colleagues providing input along the way. I did this when I was a researcher at the National Research Council Canada. Thanks! :-)

I just realized that the pre-print of the paper is available at the NRC's Publications Archive: https://nrc-publications.canada.ca/eng/view/object/?id=63e86...

Re: Classifying all of the pdfs on the internet

#114

Post by Loughla »

a123b456c wrote: post #111
j_bum wrote: post #100

Earlier quoted context omitted.

In biomedical research or tangential fields, author order generally follows these guidelines: First author(s): the individual(s) who organized and conducted the study. Typically there is only a single first author, but nowadays there are often two first authors. This is because the amount of research required to generate “high impact” publications simply can’t be done by a single person. Typically, the first author i…

Agree with this but it does not apply to all fields. Economists have a norm of alphabetizing author names unless the contributions were very unevenly distributed. That way authors can avoid squabbling over contributions.

That's how education is too. That's why I asked; it wasn't in alphabetical order.

Re: Classifying all of the pdfs on the internet

#115

Post by dangoodmanUT »

snats wrote: post #55

Hi! Author here, I wasn't expecting this to be at the top of HN, AMA

Great post, I'm wondering if there are resources you'd suggest to learn this kind of analysis?

I dug through the code and it seemed like a ton of things I'm not familiar with, probably a lot of techniques I don't know rather than python ofc.

Re: Classifying all of the pdfs on the internet

#116

Post by j_bum »

Loughla wrote: post #114
a123b456c wrote: post #111

Earlier quoted context omitted.

Agree with this but it does not apply to all fields. Economists have a norm of alphabetizing author names unless the contributions were very unevenly distributed. That way authors can avoid squabbling over contributions.

That's how education is too. That's why I asked; it wasn't in alphabetical order.

Interesting!

I always have wondered with these approaches, is there anything in the paper that indicates who was the “lead” author?

Also, to me, the alphabetical order approach reinforces issues with lower rank last names having various advantages. E.g, lower rank alphabetical names doing better in school [0]. Do you have any counterpoint to this that I’m missing?

[0] https://news.umich.edu/keeping-up-with-the-joneses-when-it-c...

Re: Classifying all of the pdfs on the internet

#117

Post by namrog84 »

CaptainFever wrote: post #90
loa_in_ wrote: post #84

Earlier quoted context omitted.

It doesn't take away the torrents, no?

One of my pet peeves is the way people use words like slurp, hoover, take, vaccuum, suck up, or steal, when in reality they mean copy. I mean if Chegg manages to sell something you can get for free, then all the more power to them lol. Though we could probably do more to educate the younger generation on the magic of torrents. Ignoring angry textbook publishers, of course.

If they copy then do things to help take down the initial thing. They've done more than just copy. And that's what I believe many places did.

You get your copy. Then send out fake dmca noticed or buy out places. Then sell your copy after everyone else's copies aren't available anymore.

It's a very standard practice in all walks of life. You gain access to a device then improve security and fix issues so that others can't get in anymore. That's a bad person approach. The legal way is to pull up the proverbial ladder or legal loopholes behind you. Good, bad, or whatever. Tons of people and places do it.

It's far more nefarious than just so innocently copying

Post reply on HN