Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Classifying all of the pdfs on the internet

snats.xyz

Post reply on HN
Print view View original

41–50 of 117 posts

Re: Classifying all of the pdfs on the internet

#41

Post by Propelloni »

IggleSniggle wrote: post #7
Propelloni wrote: post #3

Earlier quoted context omitted.

Doesn't sound like a lot, but where I am now we routinely work on very large infrastructure projects and the plans, documents and stuff mostly come as PDF. We are talking of thousands of documents, often with thousands of pages, per project and even very big projects almost never break 20 GB. If you like, you could say, PDF are information dense, but data sparse. After all it is mostly white space ;)

They often aren't like you're describing, though. For example, pdfs with high res images embedded that are drafts of future book or pamphlets prints. These can be hundreds of Mbs for a single pdf with less than 100 pages, and are so common in marketing departments that it's hard to imagine that you could fit anywhere close to all the pdfs on 8TB.

True, we get plenty of high-res pictures of film in PDF here and some of them are ridiculously large, easily approaching gigabyte sizes, like you said. But that's more a problem of the user creating the PDF than inherent to PDFs. A raw 36 megapixels (our fancy 4K displays are only 8.3 megapixels, for comparison) picture reproduction of an ISO 400 film takes only about 70 MB, which tells us that something went wrong in the transfer if a PDF containing 10 pages of them cracks 1 GB.

So, yeah, there are these monsters that send even beefy computers thrashing. But in my experience something in the creation process went wrong and it is appallingly common for a trade where PDFs are the go-to transfer format (I'm looking at you AutoCAD users!) I'd guess that the archive is doing the same we do, reprocess them for sensible results and store them. I assume you think the archive does not and then I'd agree with you. One determined civil engineer with AutoCAD can fill 8 TB in a week ;)

Re: Classifying all of the pdfs on the internet

#42

Post by Maxion »

tylerflick wrote: post #23
buildbot wrote: post #18

Earlier quoted context omitted.

No torrents at all in this data, all publicly available/open access. Mostly scientific pdfs, and a good portion of those are scans not just text. So the actual text amount is probably pretty low compared to the total. But still, a lot more than 8TB of raw data out there. I bet the total number of PDFs is close to a petabyte if not more.

> I bet the total number of PDFs is close to a petabyte if not more. That's a safe bet. I'v seen PDF's in the GBs from users treating it like a container format (which it is).

It's probably tens of petabytes if not more, if you count PDFs that'd be private. Invoices, order confirmations, contracts. There's just so so much.

Re: Classifying all of the pdfs on the internet

#43

Post by whistle650 »

Interesting read with lots of good detail, thank you. A comment: if you are balancing the classes when you do one vs all binary training, and then use the max probability for inference, your probabilities might not be calibrated well, which could be a problem. Do you correct the probabilities before taking the argmax?

Re: Classifying all of the pdfs on the internet

#44

Post by minimaxir »

One of the now-underdiscussed features of embeddings is that you can indeed use any existing statistical modeling techniques on them out of the box, and as a bonus avoid the common NLP preprocessing nuances and pitfalls (e.g. stemming) entirely.

This post is a good example on why going straight to LLM embeddings for NLP is a pragmatic first step, especially for long documents.

Re: Classifying all of the pdfs on the internet

#45

Post by buildbot »

reaperducer wrote: post #36
sporedro wrote: post #14

Earlier quoted context omitted.

Just wondering what do you collect? Is it mainly mirroring things like libgen? I have a decent collection of ebooks/pdfs/manga from reading. But I can’t imagine how large a 20TB library is.

Just wondering what do you collect? I can't speak for the OP, but you can buy optical media of old out-of-print magazines scanned as PDFs. I bought the entirety of Desert Magazine from 1937-1985. It arrived on something like 15 CD-ROMS. I drag-and-dropped the entire collection into iBooks, and read them when I'm on the train. (Yes, they're probably on archive.org for free, but this is far easier and more convenient,…

Yep, a good bit of them are from sources like this :)

Re: Classifying all of the pdfs on the internet

#46

Post by SnowflakeOnIce »

afh1 wrote: post #2

Interesting read, I did not know about Common Crawl. I feel like RTBF is kind of a lost battle these days with more and more crawlers for AI and whatnot. Once on the internet there is no way back, for better or for worse. This tangent aside, 8TB is really not a lot of data, it's just 8 consumer-grade 1TB hard drives. I find it hard to believe this is "the largest corpus of PDFs online", maybe the largest public one.…

The common crawl only pulls documents less than a small limit (1MiB last I checked). Without special handling in this project, bigger documents than that would be missing.

So indeed, not representative of the whole Internet.

Re: Classifying all of the pdfs on the internet

#47

Post by guiomie »

Interesting and fun article! I've been experimenting with various LLMs/GenAI solutions to extract tabular data from PDFs with underwhelming results. It seems like they are good at extracting strings of text and summarizing (e.g what was the total price? when was this printed?) but extracting reliably into a CSV has a decent margin of error.

Re: Classifying all of the pdfs on the internet

#48

Post by buildbot »

mehulashah wrote: post #32
buildbot wrote: post #11

I have 20-40TB (pre-dedup) of PDFs - 8TB is a lot but not even close to the total number of PDFs available.

Care to make it publicly available? Or is that not permitted on your dataset? Certainly, there’s a lot more PDFs out there than 8TB. I bet there’s a lot of redundancy in yours, but doesn’t dedup well because of all the images.

I think that would be legally iffy for the stuff like collections of old magazines that were purchased on CD/DVD and such :/

Re: Classifying all of the pdfs on the internet

#49

Post by ziddoap »

SnowflakeOnIce wrote: post #46
afh1 wrote: post #2

Interesting read, I did not know about Common Crawl. I feel like RTBF is kind of a lost battle these days with more and more crawlers for AI and whatnot. Once on the internet there is no way back, for better or for worse. This tangent aside, 8TB is really not a lot of data, it's just 8 consumer-grade 1TB hard drives. I find it hard to believe this is "the largest corpus of PDFs online", maybe the largest public one.…

The common crawl only pulls documents less than a small limit (1MiB last I checked). Without special handling in this project, bigger documents than that would be missing. So indeed, not representative of the whole Internet.

From the article:

>Specifically, when Common Crawl gets to a pdf, it just stores the first megabyte of information and truncates the rest.

This is where SafeDocs or CC-MAIN-2021-31-PDF-UNTRUNCATED enters the picture. This corpus was originally created by the DARPA SafeDocs program and what it did was refetch all the different pdfs from a snapshot of Common Crawl to have untruncated versions of them.

Re: Classifying all of the pdfs on the internet

#50

Post by daemonologist »

Propelloni wrote: post #3
afh1 wrote: post #2

Interesting read, I did not know about Common Crawl. I feel like RTBF is kind of a lost battle these days with more and more crawlers for AI and whatnot. Once on the internet there is no way back, for better or for worse. This tangent aside, 8TB is really not a lot of data, it's just 8 consumer-grade 1TB hard drives. I find it hard to believe this is "the largest corpus of PDFs online", maybe the largest public one.…

Doesn't sound like a lot, but where I am now we routinely work on very large infrastructure projects and the plans, documents and stuff mostly come as PDF. We are talking of thousands of documents, often with thousands of pages, per project and even very big projects almost never break 20 GB. If you like, you could say, PDF are information dense, but data sparse. After all it is mostly white space ;)

I'm doing some work for a company that handles scanned documents (PDFs which are purely images) and they accumulate about 15 TB / year. Of course the actual amount of information is relatively small, just inflated by being scanned. Probably 80% of them were typed up, printed, and then scanned or faxed, and of course the first thing we do is OCR them to try to recover the original text and formatting...
Post reply on HN