What's so hard about PDF text extraction? • Hacker News Board

Re: What's so hard about PDF text extraction?

#221

Post by BlueTemplar » Tue, Mar 03, 2020, 9:33 PM UTC

malcolmhere wrote: post #198

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

I've written similar code for investment banks, to extract financial reporting data from PDFs. It's shocking to think how much of the financial world runs on this kind of tin-cans-on-a-piece-of-string solution.

Do these pdfs even get printed ?

Re: What's so hard about PDF text extraction?

#222

Post by bhl » Tue, Mar 03, 2020, 9:36 PM UTC

the_french wrote: post #47

Is there a tool that works for the limited subset of PDFs generated by Latex? Do those documents have more structure than the average PDF? Less? It'd be nice to extract text from scientific articles at least.

Not a general tool but arxiv-vanity - which produces webpages of articles submitted to arxiv - works by parsing the source code that's submitted along with the PDF. You can probably use this data to train a model that converts between pdf, tex, and html.

Re: What's so hard about PDF text extraction?

#223

Post by BlueTemplar » Tue, Mar 03, 2020, 9:43 PM UTC

goatlover wrote: post #102

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

It was never meant to be a data storage format. It's for reading and printing.

Except it sucks for reading?

Re: What's so hard about PDF text extraction?

#224

Post by jorgenveisdal » Tue, Mar 03, 2020, 9:48 PM UTC

So true

Re: What's so hard about PDF text extraction?

#225

Post by Defenestresque » Tue, Mar 03, 2020, 9:51 PM UTC

siftrics wrote: post #211

ccozan wrote: post #207

Earlier quoted context omitted.

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

Your website demo video is impressive and I can imagine there are many businesses that would save a lot of time and man-hours by incorporating a solution like this.

I've often thought about creating products like these but as a one-man operation I am daunted by the "getting customers" part of the endeavour. How do you get a product like this into the hands of people who make the decisions in a business? (For anyone, not just OP). PPC AdWords campaigns? Cold-calling? Networking your ass off? Pay someone? Basically, how does one solve the "discoverability problem"?

Re: What's so hard about PDF text extraction?

#226

Post by gnicholas » Tue, Mar 03, 2020, 9:55 PM UTC

The article mentions various ways in which text that appears normal can actually be screwed up inside a PDF. I have found this when running PDFs through the BeeLine Reader PDF converter that my startup built.

One workaround I've found is that sometimes it helps to "print to PDF" the original PDF using Preview on Mac. This doesn't fix all the problems, but it does sometimes fix issues with the input PDF — even though both files appear identical to the human eye.

Are there any other workarounds or "PDF cleaners" out there? It would be awesome if there were a web-based service where you could get a PDF de-gunkified, for lack of a better term.

Re: What's so hard about PDF text extraction?

#227

Post by emodendroket » Tue, Mar 03, 2020, 10:00 PM UTC

pqdfdsaff wrote: post #182

gmiller123456 wrote: post #178

Earlier quoted context omitted.

Without having heard of or tested the solution, I'll bet anyone $1M that I can produce an image that produces an incorrect answer. Which would mean it's not "solved".

If I can produce an image that you incorrectly label as Bird or No Bird, does that mean it's accurate to say you cannot tell me if pictures have birds in them? Or is that needlessly pedantic beyond any practical use case and clearly the intended context?

No, I don't think it's pedantic and impractical. There are many use cases where the possibility of adversarial input makes heuristics a non-starter.

Re: What's so hard about PDF text extraction?

#228

Post by ashishb » Tue, Mar 03, 2020, 10:02 PM UTC

I faced many of the same issues while working on a side-project (decksaver.ashishb.net) and eventually went for OCR approach to extract text.

Re: What's so hard about PDF text extraction?

#229

Post by hamburglar » Tue, Mar 03, 2020, 10:06 PM UTC

Koshkin wrote: post #196

adrianN wrote: post #58

Earlier quoted context omitted.

What alternative do you propose? Postscript?

Why not, .ps.gz works pretty well.

... and is much more difficult to extract text from than PDF, given that it's turing complete (hello halting problem) and doesn't even restrict your output to a particular bounding box.

Re: What's so hard about PDF text extraction?

#230

Post by WalterBright » Tue, Mar 03, 2020, 10:10 PM UTC

> our most successful solution was to run OCR on these pages.

That's the most interesting point in the article.

Reminds me of how a friend managed to fix bugs in an assembly source file written in the original programmer's very own undocumented special language implemented in the assembler's macro language. He disassembled the resulting object file, fixed the problems, and checked in the disassembly as the new source code.