The take-away is that PDF should not be an input to anything.
Except eyeballs and printers, and printers are just an eyeball abstraction.
What's so hard about PDF text extraction?
51–60 of 350 posts
Re: What's so hard about PDF text extraction?
#52I suppose the best approach is to combine OCR techniques while taking hints from the PDF structure.
Or maybe use a text corpus to infer likely next words between 2 lines of characters so we can infer the reading order and columns using the text.
Re: What's so hard about PDF text extraction?
#53Around couple of years ago I am working on a home project and utilised Tesseract and Laptonica for OCR. Storage and search HDFS, HBase and SolrCloud on extracted text. You can find the details here on my website. I was very impressed with conversion of hand written pdf docs with 90% readable accuracy. I have named it as Content Data Store(CDS) http://ammozon.co.in/headtohead/?p=153 . Source code is open and you may f…
I had a similar problem and ended up using AWS' Textract tool to return the text as well as bounding box data for each letter, then overlayed that on a UI with an SVG of the original page, allowing the user to highlight handwritten and typed text. I plan to open source it so if anyone's interested let me know.
Not a fan of the potential vendor lock in though, so it's only really suitable for those in an already AWS environment not worried about them harvesting your data.
Re: What's so hard about PDF text extraction?
#54Earlier quoted context omitted.
> Finding an algorithm that approximates how a human approaches a page layout doesn’t feel like it would be all that hard. "In CS, it can be hard to explain the difference between the easy and the virtually impossible." https://xkcd.com/1425/
Totally agree. I worked on a project that had to try and extract tables from PDFs. It is much harder that it would first appear.
Detecting where tables are is still an active research areas. Once we know where on the page, it’s easier to parse out their structure.
Re: What's so hard about PDF text extraction?
#55The relatively small company I work for makes me fill out some forms by hand, because they receive them from vendors as a PDF. So I print it out, sign it, and return it to my company by hand. If someone could make a service that lets you upload a PDF that contains a form, and then let users fill out that form and e-sign it and collect the results, and then print them out all at once, it would be great. It's not a bil…
I use Xournal - https://sourceforge.net/projects/xournal/ It lets me type in to forms - or draw text over them if necessary. Then I paste in a scan of my signature. Then save as a PDF an email across. I've been doing this for years. Job applications, mortgages, medical questionnaires. No one has every queried it. If you're hand delivering a printed PDF, it's just going to be copy-typed by a human into a computer. No…
I used Xournal for a couple years in college. It was perfect in how simple it was to mix handwritten and typed notes or markup documents. The only thing is that I wish it had some sort of notebook organization feature. It would have been nice keeping all of my course notes in one file, broken down by chapter or daily pages. Instead, I ended up with a bunch of individual xojs that did the job but made searching for material take longer.
Re: What's so hard about PDF text extraction?
#56PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.
PDF is good at what it's supposed to be good. Parsing pdf to extract data is like using a rock as a hammer and a screw as a nail, if you try hard enough it'll eventually work but it was never intended to be used that way.
Actually, parsing text data from a pdf is more like using the rock to unscrew a screw, in that it was not meant to be done that way at all. But yeah, the pdf was designed to provide a fixed-format document that could be displayed or printed with the same output regardless of the device used.
I'm not sure (I haven't thought about it a lot) that you could come up with a format that duplicates that function and is also easier to parse or edit.
Re: What's so hard about PDF text extraction?
#57PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.
That’s the approach I’m using to reformat “reflow” PDFs for mobile in my app https://readerview.app/
I have been waiting for this for so long. It really works, well done.
Re: What's so hard about PDF text extraction?
#58PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.
What alternative do you propose? Postscript?
Re: What's so hard about PDF text extraction?
#59PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.
PDF is great what it meant to be, a digital printed paper, with its pros (It will look exactly the same anywhere) and cons (Can't easily extract data from it or modify it).
Currently, there is no viable alternative if you want the pros but not the cons
Re: What's so hard about PDF text extraction?
#60I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…
Maybe there's a SasS opportunity for you to explore.
I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down.
The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing.
Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice.
Off course there were workarounds, but at some point it became unmaintanable.