What's so hard about PDF text extraction? • Hacker News Board

Re: What's so hard about PDF text extraction?

#251

Post by Defenestresque » Tue, Mar 03, 2020, 11:41 PM UTC

robinhowlett wrote: post #107

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

Thanks for the links - agree about the (x,y,text) callout but other metadata like font size can be useful too. Regexes have limitations but I was able them to leverage them sufficiently for PDFs from a single source. I parsed over 1 million PDFs that had a fairly complex layout using Apache PDFBox and wrote about it here: https://www.robinhowlett.com/blog/2019/11/29/parsing-structu...

I thoroughly enjoyed both the blog post (as an accessible but thorough explanation of your experience with PDF data extraction) and the linked news article [0] as an all-too-familiar story of a company realizing that a creative person is using their freely-available data in novel and exciting ways and immediately requesting that they shut it down, because faced with the perceived dichotomy of maintaining control versus encouraging progress they will often play on the safe side.

[0] https://www.thoroughbreddailynews.com/getting-from-cease-and...

Re: What's so hard about PDF text extraction?

#252

Post by perl4ever » Tue, Mar 03, 2020, 11:44 PM UTC

Alex3917 wrote: post #179

tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Since you can always print a PDF to a bitmap and use OCR, I assume you're implicitly asking for something that does substantially better. How much better, and why?

Re: What's so hard about PDF text extraction?

#253

Post by 72deluxe » Tue, Mar 03, 2020, 11:46 PM UTC

dwheeler wrote: post #5

The best technique for having a PDF with extractable data is to include the data within the PDF itself. That is what LibreOffice can do, it can slip in the entire original document within a PDF. Since a compressed file is quite small, the resulting files are not that much larger, and then you don't need to fuss with OCR or anything else.

How does LibreOffice include the entire document with the PDF?

Is there a special "data" section of the PDF that includes this? Can you point me to any documentation regarding this? It sounds quite good TBH.

Re: What's so hard about PDF text extraction?

#254

Post by 72deluxe » Tue, Mar 03, 2020, 11:49 PM UTC

hnick wrote: post #246

mikestew wrote: post #51

Earlier quoted context omitted.

Except eyeballs and printers, and printers are just an eyeball abstraction.

You'd hope so, but some printers run some very finicky software with less horsepower than your desktop machine so can fall over on complex PDF structures. I preferred Postscript!

How does PCL cope with PDFs? Do you know if some conversion happens beforehand?

Re: What's so hard about PDF text extraction?

#255

Post by Rury » Tue, Mar 03, 2020, 11:53 PM UTC

equasar wrote: post #157

bob1029 wrote: post #69

Earlier quoted context omitted.

We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…

Any recommended library for .NET to extract text by coordinates?

There's itext7 (also for java). Not sure how it compares with other libraries, but it will parse text along with coordinates. You just need to write your own execution strategy to parse how you want.

From my experience, it seems to grab text just fine, the tricky part is identifying & grabbing what you want, and ignoring what you don't want... (for reasons mentioned in the article)

https://github.com/itext/itext7-dotnet

https://itextpdf.com/en/resources/examples/itext-7/parsing-p...

Re: What's so hard about PDF text extraction?

#256

Post by 72deluxe » Tue, Mar 03, 2020, 11:54 PM UTC

aasasd wrote: post #180

mLuby wrote: post #165

Earlier quoted context omitted.

Not everything needs to look good on every screen size. I don't expect to be able to read academic papers on my smartwatch, and a simple alarm clock app looks kind of silly when it's fullscreen across a desktop monitor. Likewise, when layout makes the difference between reader understanding or confusion, it's hard to trust automatic reflowing on unknown screen sizes. PDF is simply better than HTML when it comes to pr…

I wonder if you realize that both your points wildly miss what I said. First, there's no need to stretch my argument to the point of it being ridiculous. I don't have to reach for a watch to suffer from PDF. Even a tablet is enough: I don't see many 14" tablets flying off the shelves. I also know for sure that the vast majority of ubiquitous communicator devices , aka smartphones, are about the same size as mine, so…

Some people are very "funny" about the layout of items and text and want it to be preserved identically to their "vision" when they created it. For example, every "marketing" individual when they see a webpage seem to want it pixel-perfect.

I think it's the artist in them.

This is understandable in some instances:

a. Picasso's or Monet's works probably wouldn't be as good if you just roll them up into a ball. Sure, the component parts are still there (it's just paper/canvas and paint after all!) but the result isn't what they intended.

b. A car that has hit a tree is made up of the composite parts but isn't quite as appealing (or useful) as the car before hitting the tree.

c. A wedding cake doesn't look as good if the ingredients are just thrown over the wedding party's table. The ingredients are there, but it just isn't the same...

Presentation is sometimes important.

Re: What's so hard about PDF text extraction?

#257

Post by 72deluxe » Tue, Mar 03, 2020, 11:57 PM UTC

mqus wrote: post #126

hwc wrote: post #24

I worked on PDF generating software for years. It's a horrible format that should never have been approved as an ISO standard. When in doubt, use plain text. It's a million times better in every way that counts. I wish my bank statements and such could be downloaded as plain text files, instead of massive PDF files that embed another copy of a bunch of typefaces in each file.

Since gdpr, businesses are "required" to make your data available to you for transfer in a machine-readable format and you could argue that pdf is not exactly machine-readable in the sense of the law. Practically I have seen cases where you do get csv or something similar, but especially smaller firms will probably give you word documents, excel files or pdfs.

Interesting! Halifax Bank in the UK changed their generation library for PDFs the other year such that new statements rendered incorrectly on the Mac. Old statements were fine. New ones were garbage text.

Chrome displayed them fine, Preview on Mac did not.

Trying to communicate this to them was like talking to a tree, or an alien, or a room of catatonic individuals.

Thankfully I think they've fixed it now.

Re: What's so hard about PDF text extraction?

#258

Post by hnick » Tue, Mar 03, 2020, 11:58 PM UTC

72deluxe wrote: post #254

hnick wrote: post #246

Earlier quoted context omitted.

You'd hope so, but some printers run some very finicky software with less horsepower than your desktop machine so can fall over on complex PDF structures. I preferred Postscript!

How does PCL cope with PDFs? Do you know if some conversion happens beforehand?

Not sure if I'm misunderstanding but PCL is another page description language like PS, some printers can use both depending on the driver.

Most of our Xerox printers spoke Postscript natively, these days more printers can use PDF. We generally used a tool to convert PCL to PS to suit our workflow if that was the only option for the file, because being able to manipulate the file (reordering and applying barcodes or minor text modifications) was important. Likewise for AFP and other formats. PCL jobs were rare so I never worked on them personally.

Re: What's so hard about PDF text extraction?

#259

Post by 72deluxe » Wed, Mar 04, 2020, 12:00 AM UTC

Jaruzel wrote: post #100

garethl wrote: post #82

The open-source Ghostscript [1] can convert simple PDFs to text, while keeping the layout. I doubt it will handle some of the more complicated cases outlined in the article though. I use it quite successfully to turn my bank statements into text, which can then be further processed. [1]: https://www.ghostscript.com/

I've recently done this. Have scanned over 5,000 documents to PDF, then batch converted those from PDF to TIFF using Ghostscript, and then Tesseract to OCR the TIFF and combine both back into a searchable PDF. Tesseract may not be the worlds best OCR software but it's free and both it and Ghostscript are easy to automate. Now all I need is a good front end search system for my document archive.

How did you scan the documents to PDF? I use a Canon P-208 that has served me well for many many years (long may it!) and the OCR on that works well.

Does the scanning system you use not do OCR?

I use a Mac and Spotlight does a good job of indexing the files. I think alternatives for other OSes might be something like Apache Solr?

Re: What's so hard about PDF text extraction?

#260

Post by abiogenesis » Wed, Mar 04, 2020, 12:00 AM UTC

rbobby wrote: post #241

tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> double-tap to zoom Why wouldn't you just zoom with the center point being where the tap occurred?

That's probably what they did for v1.0 after they saw it is not that easy to zoom such that the whole paragraph always fits into the visible area.