Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

131–140 of 350 posts

Re: What's so hard about PDF text extraction?

#131

Post by mtlogstdo »

pacoverdi wrote: post #78
daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Many years ago, I regularly had to parse specifications of protocols from various electronic exchanges. The general approach I used was to do a first pass using a Linux tool to convert it to text: pdftotext. Something like: pdftotext -layout -nopgbrk -eol unix -f $firstpage -l $lastpage -y 58 -x 0 -H 741 -W 596 "$FILE" After that, it was a matter of writing and tweaking custom text parsers (in python or java) until t…

I worked for an epub firm that used a similar approach a while ago - we took PDFs and produced Flash (yes, that old) versions for online, and created iOS and Android apps for the publisher.

I've come across most of the problems in this post but the most memorable thing was when we were asked to support Arabic, when suddenly all your previous assumptions are backwards!

Re: What's so hard about PDF text extraction?

#132

Post by voicesarefree »

Wish I had this to share with my boss years ago. My first big project at my first post-college job was building a PDF parser that would generate notifications if a process document had been updated and it was the first time the logged in user was seeing it (to ensure they read the changelog of the process). Even with a single source of the PDFs (one technical document writer) I could only get a 70% success rate because the text I needed to parse was all over the place, when I stated we would need to use OCR to get better results no further development was done (ROI reasons). The technical writer was unwilling to standardize more than they already had, or consider an alternative upload process where they confirm the revision information.. which didn't help.

I don't envy working on ingesting even more diverse PDFs.

Re: What's so hard about PDF text extraction?

#133

Post by ethanwillis »

I went down a rabbit hole while making a canvas based UI library from scratch.. and started reading about the history of NeWS, display postscript, and postscript in general.

I started reading the ISO spec for postscript used in modern PDFs. You can read it yourself here: https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/PD...

What actually needs to be done to extract text correctly is to be able to parse the postscript, have a way of figuring out how the raw text.. or the curves that draw the text.. are displayed (whether they are or not and in relation to each other) using information that the postscript gives you.

Edit: More than anything I think understanding deeply the class of PDFs you want to extract data from is the most important part. Trying to generalize it is where the real difficulty comes from.. as in most things.

Re: What's so hard about PDF text extraction?

#134

Post by jazzido »

staticautomatic wrote: post #124
jazzido wrote: post #116

Earlier quoted context omitted.

Hi, author and maintainer of Tabula ( https://github.com/tabulapdf/tabula ). We've been trying to contact you about the "Tabula Pro" version that you are offering. Feel free to reachme at manuel at jazzido dot com

Edit: See reply below Am I reading the repos correctly? It looks like Extractable copied Tabula (MIT) to its own repo rather than forking it, removed the attribution, and then tried to re-license it as Apache 2.0. If so, that would be pretty fucked up. https://github.com/tabulapdf https://github.com/ExtractTable/tabulapro

Not really. They import tabula_py, which is a Python wrapper around tabula-java (the library of which I'm a maintainer).

Still, I would have loved at least a heads up from the team that sells Tabula Pro. I know they're not required to do so, but hey, they're kinda piggybacking on Tabula's "reputation".

Re: What's so hard about PDF text extraction?

#135

Post by degski »

ProZsolt wrote: post #103
mumblemumble wrote: post #62

Earlier quoted context omitted.

> Currently, there is no viable alternative if you want the pros but not the cons I remember OpenXPS being much easier to work with. That might be due to cultural rather than structural differences, mind - fewer applications generate OpenXPS, so there's fewer applications to generate them in their own special snowflake ways.

This is the first time I heard of it. When I search for it I only find the Wikipedia article and 99 links to how to convert it to pdf. The problem with this is that from an average person perspective it doesn't have the pros. There is no built-in or first-party app that can open this format on Mac and Linux. More than 99% of the users only want to read or print it. It's hard to convince them to use an alternative for…

It's a Windows-thing, since W7, IIRC. It's ok now, but it has been buggy for years, and yes, who eats xps-files, so better it is, but it's not more useful.

Re: What's so hard about PDF text extraction?

#136

Post by _rerr »

There is a fairly interesting library developed by the Stanford Team behind https://www.snorkel.org/ that takes structured documents, including PDF formatted as tables, and builds a knowledge base: https://github.com/HazyResearch/fonduer

It looks promising for these kinds of daunting tasks

Re: What's so hard about PDF text extraction?

#137

Post by staticautomatic »

jazzido wrote: post #134
staticautomatic wrote: post #124

Earlier quoted context omitted.

Edit: See reply below Am I reading the repos correctly? It looks like Extractable copied Tabula (MIT) to its own repo rather than forking it, removed the attribution, and then tried to re-license it as Apache 2.0. If so, that would be pretty fucked up. https://github.com/tabulapdf https://github.com/ExtractTable/tabulapro

Not really. They import tabula_py, which is a Python wrapper around tabula-java (the library of which I'm a maintainer). Still, I would have loved at least a heads up from the team that sells Tabula Pro. I know they're not required to do so, but hey, they're kinda piggybacking on Tabula's "reputation".

Thanks for clarifying.

Re: What's so hard about PDF text extraction?

#138

Post by alkonaut »

aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

A PDF isn’t for storage it’s for display. It’s the equivalent of a printout. You don’t delete your CAD drawing or spreadsheet after printing it out.

Re: What's so hard about PDF text extraction?

#139

Post by grishka »

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

You clearly haven't ever worked with MP3.

Re: What's so hard about PDF text extraction?

#140

Post by dredmorbius »

pacoverdi wrote: post #78
daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Many years ago, I regularly had to parse specifications of protocols from various electronic exchanges. The general approach I used was to do a first pass using a Linux tool to convert it to text: pdftotext. Something like: pdftotext -layout -nopgbrk -eol unix -f $firstpage -l $lastpage -y 58 -x 0 -H 741 -W 596 "$FILE" After that, it was a matter of writing and tweaking custom text parsers (in python or java) until t…

I've discovered page-oriented processing in awk, which is a godsend for parsing PDFs.

See:

https://news.ycombinator.com/item?id=22156456

In the GNU Awk User's Guide:

https://www.gnu.org/software/gawk/manual/html_node/Multiple-...

Tracking column and field widths across page breaks is ... interesting, but more tractable.

Post reply on HN