What's so hard about PDF text extraction? • Hacker News Board

Re: What's so hard about PDF text extraction?

#81

Post by carapace » Tue, Mar 03, 2020, 3:22 PM UTC

ProZsolt wrote: post #59

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

PDF is great what it meant to be, a digital printed paper, with its pros (It will look exactly the same anywhere) and cons (Can't easily extract data from it or modify it). Currently, there is no viable alternative if you want the pros but not the cons

Screenshots of Smalltalk. (I'm joking.)

Re: What's so hard about PDF text extraction?

#82

Post by garethl » Tue, Mar 03, 2020, 3:28 PM UTC

The open-source Ghostscript [1] can convert simple PDFs to text, while keeping the layout. I doubt it will handle some of the more complicated cases outlined in the article though.

I use it quite successfully to turn my bank statements into text, which can then be further processed.

[1]: https://www.ghostscript.com/

Re: What's so hard about PDF text extraction?

#83

Post by anthk » Tue, Mar 03, 2020, 3:30 PM UTC

Iwillgetby wrote: post #80

If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…

>- Run a jpeg to pdf tool.

ImageMagick. convert *.jpg out.pdf

Re: What's so hard about PDF text extraction?

#84

Post by sudhirj » Tue, Mar 03, 2020, 3:31 PM UTC

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Are you open to doing more of this? Trying to do the same thing but I’d rather have an expert do it and focus on the app.

Re: What's so hard about PDF text extraction?

#85

Post by rwojo » Tue, Mar 03, 2020, 3:32 PM UTC

hylian wrote: post #53

mkjmkumar wrote: post #49

Around couple of years ago I am working on a home project and utilised Tesseract and Laptonica for OCR. Storage and search HDFS, HBase and SolrCloud on extracted text. You can find the details here on my website. I was very impressed with conversion of hand written pdf docs with 90% readable accuracy. I have named it as Content Data Store(CDS) http://ammozon.co.in/headtohead/?p=153 . Source code is open and you may f…

I had a similar problem and ended up using AWS' Textract tool to return the text as well as bounding box data for each letter, then overlayed that on a UI with an SVG of the original page, allowing the user to highlight handwritten and typed text. I plan to open source it so if anyone's interested let me know. Not a fan of the potential vendor lock in though, so it's only really suitable for those in an already AWS e…

Very interested to see this as I was about to work on the same thing!

Re: What's so hard about PDF text extraction?

#86

Post by anthk » Tue, Mar 03, 2020, 3:34 PM UTC

garethl wrote: post #82

The open-source Ghostscript [1] can convert simple PDFs to text, while keeping the layout. I doubt it will handle some of the more complicated cases outlined in the article though. I use it quite successfully to turn my bank statements into text, which can then be further processed. [1]: https://www.ghostscript.com/

MuPDF and XPDF can do that by selecting the text with the right mouse buttom.

Re: What's so hard about PDF text extraction?

#87

Post by bsdubernerd » Tue, Mar 03, 2020, 3:38 PM UTC

It's nice to note how several of these problems already exist in much more structured document types, such as HTML.

Using white-on-white dark-hat SEO techniques for keyword boosting? Check. Custom fonts with random glyphs? Check. I didn't see custom encodings (yet).

We try to keep HTML semantic, but google has been interpreting pages to a much higher level in order to spot issues such as these. If you ever tried to work on a scraper, you know how it's very hard to get far nowdays without using a full-blown browser as a backend.

What worries me is that it's going to get massively worse. Despite me hating HTML/web interfaces, one big advantage for me is that everything which looks like text is normally selectable, as opposed to a standard native widget which isn't. It's just much more "usable", as a user, because everything you see can be manipulated.

We've seen already asm.js-based dynamic text layout inspired by tex with canvas rendering that has no selectable content and/or suffers from all the OP issues! Now, make it fast and popular with WASM...

"yay"

Re: What's so hard about PDF text extraction?

#88

Post by siftrics » Tue, Mar 03, 2020, 3:41 PM UTC

I’m the founder of a startup (https://siftrics.com) that’s trying to solve this problem. We’re growing extremely quickly.

Way too busy to write more, but I’ll be back to read comments later tonight.

Re: What's so hard about PDF text extraction?

#89

Post by aasasd » Tue, Mar 03, 2020, 3:41 PM UTC

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure.

And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF for precise formatting of their papers—which mostly barely differ from Markdown by having two columns and formulas. What exactly they need ‘precise formatting’ for, and why it can't be solved with MathML and image fallback, they can't say.

People feeling the urge to defend PDF might want to pick up at this point in the discussion: https://news.ycombinator.com/item?id=21454636

Re: What's so hard about PDF text extraction?

#90

Post by kelvin0 » Tue, Mar 03, 2020, 3:42 PM UTC

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Well I am putting the finishing touches on a front end that allows extracting PDF text visually. It's also able to adjust when the PDF page size vary for a given document type. Once you build the extractor for a document type, it can run on a batch of PDFs and store to Excel or Database (or any other format). I sense this tool facilitates and automates a lot of the 'dark art' you mention. Of course there are always difficult documents that don't fit exactly in the initial extraction paradigm, for those I use the big guns ...