What's so hard about PDF text extraction? • Hacker News Board

Re: What's so hard about PDF text extraction?

#41

Post by Santosh83 » Tue, Mar 03, 2020, 2:06 PM UTC

maest wrote: post #31

Santosh83 wrote: post #2

Another site that breaks the browser's back navigation. Why do so many sites do this? Do they imagine they retain user attention for longer if they break navigation? It's pretty trivial to long-press the back button or just close the tab and not come back again to your site...

Hi, author here. We've taken no intentional action to change the way the back button works - in fact, I too hate it when websites do that. Can you PM me with some details about what you're seeing? I'm having issues reporducing it with my particular setup.

Good to know! I don't believe PM is possible on hacker news so I hope you don't mind that I describe some details right here?

My browser is the latest (v73.0.1) Firefox on the latest build of Windows10. I confirmed the issue with all addons disbled so it is not an addon issue. I think I know what may be responsbile. When initially I load the page the back button works as intended for about a second. After that delay the page seems to load some resources from static.parastorage.com and www.mymobileapp.online. Once those resources are finished loading the back button does not navigate back to the HN article on the first press. Have to press once more. So I presume a script from one of those domains is responsible. Hope this helps!

Re: What's so hard about PDF text extraction?

#42

Post by dragonwriter » Tue, Mar 03, 2020, 2:08 PM UTC

oddthink wrote: post #39

hwc wrote: post #24

I worked on PDF generating software for years. It's a horrible format that should never have been approved as an ISO standard. When in doubt, use plain text. It's a million times better in every way that counts. I wish my bank statements and such could be downloaded as plain text files, instead of massive PDF files that embed another copy of a bunch of typefaces in each file.

Ugh, this. I still fail to understand how a device from 2019, even a phone, could show any rendering delay when scrolling to page 200 of a 400 page static document. I thought PDF was less programmable than PostScript, but there's still got to be some kind of non-local semantics in there.

> I thought PDF was less programmable than PostScript

It's not.

It used to be long ago, but now it has full programmability with JavaScript.

Re: What's so hard about PDF text extraction?

#43

Post by Ididntdothis » Tue, Mar 03, 2020, 2:09 PM UTC

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

Yup. I still have PTSD from a project where I needed to extract text from millions of PDFs

Re: What's so hard about PDF text extraction?

#44

Post by lm28469 » Tue, Mar 03, 2020, 2:17 PM UTC

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

PDF is good at what it's supposed to be good. Parsing pdf to extract data is like using a rock as a hammer and a screw as a nail, if you try hard enough it'll eventually work but it was never intended to be used that way.

Re: What's so hard about PDF text extraction?

#45

Post by [deleted] » Tue, Mar 03, 2020, 2:20 PM UTC

throwaway3157 wrote: post #36

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Do you have a blog? I'd enjoy reading some of your tricks. Also, how do you manage things when one of those banks decides to change the layout/format?

[deleted]

Re: What's so hard about PDF text extraction?

#46

Post by Quarrelsome » Tue, Mar 03, 2020, 2:23 PM UTC

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Any tricks for decimal points versus noise? Its a terrifying outcome and all I've got is doing statistical analysis on the data you've already got and highlighting "outliers".

Re: What's so hard about PDF text extraction?

#47

Post by the_french » Tue, Mar 03, 2020, 2:24 PM UTC

Is there a tool that works for the limited subset of PDFs generated by Latex? Do those documents have more structure than the average PDF? Less? It'd be nice to extract text from scientific articles at least.

Re: What's so hard about PDF text extraction?

#48

Post by pqdfdsaff » Tue, Mar 03, 2020, 2:24 PM UTC

jrandm wrote: post #27

peteretep wrote: post #20

So all through this I’m thinking “just OCR it and be done”, and we get to: > Why not OCR all the time? > Running OCR on a PDF scan usually takes at least an order of magnitude longer than extracting the text directly from the PDF. ... so? Google can OCR video and translate it in something that feels like real-time; what PDF processing are they doing that is so performance bound? > Difficulties with non-standard chara…

> Finding an algorithm that approximates how a human approaches a page layout doesn’t feel like it would be all that hard. "In CS, it can be hard to explain the difference between the easy and the virtually impossible." https://xkcd.com/1425/

To be fair, that particular comic's 5 year task was solved shortly after it was posted, at least in part to prove it was wrong.

Re: What's so hard about PDF text extraction?

#49

Post by mkjmkumar » Tue, Mar 03, 2020, 2:28 PM UTC

Around couple of years ago I am working on a home project and utilised Tesseract and Laptonica for OCR. Storage and search HDFS, HBase and SolrCloud on extracted text. You can find the details here on my website. I was very impressed with conversion of hand written pdf docs with 90% readable accuracy. I have named it as Content Data Store(CDS) http://ammozon.co.in/headtohead/?p=153 . Source code is open and you may find steps on installation and how to run here. http://ammozon.co.in/headtohead/?p=129 http://ammozon.co.in/headtohead/?p=126 A short demo http://ammozon.co.in/gif/ocr.gif

I didnot get time to enhance it further but planning to containerize the whole application. See if you find it useful in its current form.

Re: What's so hard about PDF text extraction?

#50

Post by ldenoue » Tue, Mar 03, 2020, 2:29 PM UTC

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

That’s the approach I’m using to reformat “reflow” PDFs for mobile in my app https://readerview.app/