Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

181–190 of 350 posts

Re: What's so hard about PDF text extraction?

#181

Post by hackcasual »

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

I worked on an online retailer's book scan ingestion pipeline. It's funny because we soon got most of our "scans" as print-ready PDFs, but we still ran them through the OCR pipeline (that would use the underlying pdf text) since parsing it any other way was a small nightmare.

Re: What's so hard about PDF text extraction?

#182

Post by pqdfdsaff »

gmiller123456 wrote: post #178
pqdfdsaff wrote: post #48

Earlier quoted context omitted.

To be fair, that particular comic's 5 year task was solved shortly after it was posted, at least in part to prove it was wrong.

Without having heard of or tested the solution, I'll bet anyone $1M that I can produce an image that produces an incorrect answer. Which would mean it's not "solved".

If I can produce an image that you incorrectly label as Bird or No Bird, does that mean it's accurate to say you cannot tell me if pictures have birds in them? Or is that needlessly pedantic beyond any practical use case and clearly the intended context?

Re: What's so hard about PDF text extraction?

#183

Post by mark-r »

Finnucane wrote: post #56
lm28469 wrote: post #44

Earlier quoted context omitted.

PDF is good at what it's supposed to be good. Parsing pdf to extract data is like using a rock as a hammer and a screw as a nail, if you try hard enough it'll eventually work but it was never intended to be used that way.

Actually, parsing text data from a pdf is more like using the rock to unscrew a screw, in that it was not meant to be done that way at all. But yeah, the pdf was designed to provide a fixed-format document that could be displayed or printed with the same output regardless of the device used. I'm not sure (I haven't thought about it a lot) that you could come up with a format that duplicates that function and is also…

I would think any word processing document format would duplicate that function and be better.

Re: What's so hard about PDF text extraction?

#184

Post by pierre »

giovannibonetti wrote: post #108
robinhowlett wrote: post #107

Earlier quoted context omitted.

Thanks for the links - agree about the (x,y,text) callout but other metadata like font size can be useful too. Regexes have limitations but I was able them to leverage them sufficiently for PDFs from a single source. I parsed over 1 million PDFs that had a fairly complex layout using Apache PDFBox and wrote about it here: https://www.robinhowlett.com/blog/2019/11/29/parsing-structu...

Oh, yeah, pdf2json returns font sizes as well. I forgot to mention that.

pdf2json font name can be uncorrect sometime as it does only extract them based on a pre-set collection of fonts. I suggest using this fork that fix it :

https://github.com/AXATechLab/pdf2json

Bounding box also can be off with pdf2json. Pdf.js do a better job but have a tendency to no handling some ligature/glyph well, transforming word like finish to "f nish" sometime (eating the i in this case). pdfminer (python) is the best solution yet but a thousand time slower....

Re: What's so hard about PDF text extraction?

#185

Post by rushabhasheth »

anodyne33 wrote: post #109

Does anybody regularly use Acrobat's text extraction engine? I've had fine results as far as accuracy goes when compared to other OCR engines but one sticking point drives me nuts. My problem is, and I'm typically doing this in batches of thousands of files, if a PDF has a footer applied Acrobat sees that as renderable text and blows off the rest of the rest of the page. I've tried all manner of sanitizing, removing…

Anodyne, you can check docsumo dot com . We might be able to solve your problem

Re: What's so hard about PDF text extraction?

#186

Post by jimmaswell »

Alex3917 wrote: post #179
tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Rendering the document to an image and using OCR on it would bypass a lot of trouble trying to make sense of the source, wouldn't it?

Re: What's so hard about PDF text extraction?

#187

Post by tolmasky »

Alex3917 wrote: post #179
tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Re: What's so hard about PDF text extraction?

#188

Post by Alex3917 »

jimmaswell wrote: post #186
Alex3917 wrote: post #179

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Rendering the document to an image and using OCR on it would bypass a lot of trouble trying to make sense of the source, wouldn't it?

Not really, it's just a different set of challenges. The original article sums it up well, in terms of a lack of text-order hints. I haven't really tried incorporating OCR approaches at all, but I suspect they could probably be used to detect hidden text.

The basic issue imho is that NLP algorithms are very inaccurate even with perfect input. E.g. even with perfect input, they're maybe only 75% accurate. And even an a text-processing algorithm that's like 99.9% accurate will yield input to your NLP algorithms that's like 50% accurate, so any results will be mostly unusable.

Re: What's so hard about PDF text extraction?

#189

Post by wpietri »

jazzido wrote: post #134
staticautomatic wrote: post #124

Earlier quoted context omitted.

Edit: See reply below Am I reading the repos correctly? It looks like Extractable copied Tabula (MIT) to its own repo rather than forking it, removed the attribution, and then tried to re-license it as Apache 2.0. If so, that would be pretty fucked up. https://github.com/tabulapdf https://github.com/ExtractTable/tabulapro

Not really. They import tabula_py, which is a Python wrapper around tabula-java (the library of which I'm a maintainer). Still, I would have loved at least a heads up from the team that sells Tabula Pro. I know they're not required to do so, but hey, they're kinda piggybacking on Tabula's "reputation".

You're being much more polite here than I would be. Even if it isn't illegal, what they've done is a giant dick move.

Re: What's so hard about PDF text extraction?

#190

Post by lwhsiao »

_rerr wrote: post #136

There is a fairly interesting library developed by the Stanford Team behind https://www.snorkel.org/ that takes structured documents, including PDF formatted as tables, and builds a knowledge base: https://github.com/HazyResearch/fonduer It looks promising for these kinds of daunting tasks

One of the co-authors of Fonduer here. Just for reference the original paper for Fonduer is here:

https://dl.acm.org/doi/pdf/10.1145/3183713.3183729

And additional follow-up work on extracting data from PDF datasheets is here:

https://dl.acm.org/doi/pdf/10.1145/3316482.3326344

One thing to point out about our library is that while we do take PDF as input and use it to calculate visual features, we also rely on an HTML representation of the PDF for structural cues. In our pipeline this is typically done by using Adobe Acrobat to generate an HTML representation for each input PDF.

Post reply on HN