Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Classifying all of the pdfs on the internet

snats.xyz

Post reply on HN
Print view View original

101–110 of 117 posts

Re: Classifying all of the pdfs on the internet

#101

Post by j_bum »

gnewton77 wrote: post #63

Did some similar work with similar visualizations ~2009, on ~5.7M research articles (PDFs, private corpus) from scientific publishers Elsevier, Springer: Newton, G., A. Callahan & M. Dumontier. 2009. Semantic Journal Mapping for Search Visualization in a Large Scale Article Digital Library. Second Workshop on Very Large Digital Libraries at the European Conference on Digital Libraries (ECDL) 2009. https://lekythos.li…

Nice article, thanks for sharing.

I can imagine mining all of these articles was a ton of work. I’d be curious to know how quickly the computation could be done today vs. the 13 hour 2009 benchmark :)

Nowadays people would be slamming those data through UMAP!

Re: Classifying all of the pdfs on the internet

#102

Post by mkl »

throw10920 wrote: post #99
mkl wrote: post #98

Earlier quoted context omitted.

You can apply statistical techniques to anything you want. Embeddings are just vectors of numbers which capture some meaning, so statistical analysis of them will work fine.

Don't most statistical techniques rely on specific structure in the spaces containing the objects they operate on, in order to be useful?

Embeddings have structure, or they wouldn't be very useful. E.g. cosine similarity works because (many) embeddings are designed to support it.

Re: Classifying all of the pdfs on the internet

#103

Post by throw10920 »

mkl wrote: post #102
throw10920 wrote: post #99

Earlier quoted context omitted.

Don't most statistical techniques rely on specific structure in the spaces containing the objects they operate on, in order to be useful?

Embeddings have structure, or they wouldn't be very useful. E.g. cosine similarity works because (many) embeddings are designed to support it.

Oh, that should have been obvious. Thank you for explaining.

Re: Classifying all of the pdfs on the internet

#104

Post by zo1 »

moralestapia wrote: post #21
mellosouls wrote: post #13

Earlier quoted context omitted.

I haven't downvoted you but it is presumably because of your hasty typing or lack of proofreading/research. 33TB (first google result from 5 years ago) not 33GB. Larger figures from more recently.

>hasty typing or lack of proofreading/research This is exactly what I meant with "HN is becoming quite hostile" * I brought up something I looked up to support GP's argument. * The argument is correct. * I do it in good faith. * G is literally next to T. * I even praise the article, while at it. "Oh, but you made a typo!". Good luck, guys. I'm out. PS. I will give my whole 7 figure net worth, no questions asked, tran…

Some days it's worth it to burn some imaginary internet points for the good of the discussion and article. People downvote for various reasons, which we will never be able to figure out why definitely. Each person is different, and they all have days where they swing one way or another.

Re: Classifying all of the pdfs on the internet

#106

Post by fsckboy »

specialist wrote: post #81
fsckboy wrote: post #57

Earlier quoted context omitted.

> RTBF isn't about having your information wiped from the internet. your take is misleading enough to be considered wrong. It's "don't use public information about me in search engines, I don't want people to find that information about me", not simply "don't use my information for marketing purposes" https://en.wikipedia.org/wiki/Right_to_be_forgotten first paragraph of the article: The right to be forgotten (RTBF)…

Once demographic data cannot be crawled or cached by 3rd parties, we get RTBF for free.

RTBF does not ban crawling or caching. It bans opening up those archives to the public via search engines.

Re: Classifying all of the pdfs on the internet

#107

Post by sireat »

Nice work! You've taken multiple approaches similar to what I sometimes do at the national library, I've used all kind of embeddings -> classifiers / LDA.

Curious on your prompt: https://github.com/snat-s/m/blob/main/classify_metadata/prom...

Wouldn't this be basically prompting to classify by the type of URL?

Re: Classifying all of the pdfs on the internet

#109

Post by qingcharles »

mehulashah wrote: post #32
buildbot wrote: post #11

I have 20-40TB (pre-dedup) of PDFs - 8TB is a lot but not even close to the total number of PDFs available.

Care to make it publicly available? Or is that not permitted on your dataset? Certainly, there’s a lot more PDFs out there than 8TB. I bet there’s a lot of redundancy in yours, but doesn’t dedup well because of all the images.

I have >10TB of magazines I've collected so far, and I could probably source another 50TB if I had the time. I'm working on uploading them, but I've had too much on my plate lately: https://en.magazedia.wiki/

There is a significant issue with copyright, though. I'll remove anything with a valid DMCA, but 99.9% of the world's historical magazine issues are now in IP limbo as their ownership is probably unknown. Most of the other .1% aren't overly concerned as distribution is their goal and their main income is advertising, not sales.

Re: Classifying all of the pdfs on the internet

#110

Post by gsck »

fsckboy wrote: post #106
specialist wrote: post #81

Earlier quoted context omitted.

Once demographic data cannot be crawled or cached by 3rd parties, we get RTBF for free.

RTBF does not ban crawling or caching. It bans opening up those archives to the public via search engines.

It bans having them in the first place. Not just looking at them.
Post reply on HN