While it’s specifically Li t for PDF tables, I’ve found Tabula to be pretty robust. I use it to convert banking statements to CSV.
It’s still a pretty manual process, but it does the most difficult part good enough.
What's so hard about PDF text extraction?
11–20 of 350 posts
Re: What's so hard about PDF text extraction?
#12The relatively small company I work for makes me fill out some forms by hand, because they receive them from vendors as a PDF. So I print it out, sign it, and return it to my company by hand. If someone could make a service that lets you upload a PDF that contains a form, and then let users fill out that form and e-sign it and collect the results, and then print them out all at once, it would be great. It's not a bil…
Coming from a documents format world (publishing), there are a lot of cases like this.
In theory it sounds like it should be straightforward but it hinges so much on how well the document is structured underneath the surface.
Being that these tools were primarily designed for non-technical users first the priority is in the visual and printed outcome and not the underlying structure.
One document can look much the same as another in form—uses black borders to outline fields, similar or same field names, etc, but may be structured entirely differently and that can be a madhouse of frustrating problems.
It can be complex enough to write a solution for one specific document source. Writing a universal tool that could take in any form like that would probably be a pretty decent moneymaker.
My first intuition, though, would be it may be more successful (though no less simple) to develop a model that can read from the visual of the document rather than parsing it successfully.
Open to learning something here, though!
Re: What's so hard about PDF text extraction?
#13The relatively small company I work for makes me fill out some forms by hand, because they receive them from vendors as a PDF. So I print it out, sign it, and return it to my company by hand. If someone could make a service that lets you upload a PDF that contains a form, and then let users fill out that form and e-sign it and collect the results, and then print them out all at once, it would be great. It's not a bil…
Would DocuSign work? I’ve signed for lease documents several times that way.
Something like that would work for signing, but the hard part is "turn this pdf into an online form". That way after a user finishes a form, you can perform some basic error checking like, did they fill out everything, is this field a valid format, etc. After 100 employees turn in a multi-page printed out form, someone has to go through it and make sure they signed everywhere, filled out all the fields, etc.
Again, not sexy, but it is so stupid I have to fill out a direct deposit form by hand and turn it into my company, who checks it, then hands it off to the payroll vendor, who has to check it, just to enter the damn data into a form on their end.
Re: What's so hard about PDF text extraction?
#14On a personal project, I had a good experience extracting PDF text using Tabula[1]. You specify the bounding boxes where desired data is, and it spits out the content it finds.
It still hits the issues mentioned in this article (surprise spaces appearing in middle of words, etc)
Re: What's so hard about PDF text extraction?
#15The relatively small company I work for makes me fill out some forms by hand, because they receive them from vendors as a PDF. So I print it out, sign it, and return it to my company by hand. If someone could make a service that lets you upload a PDF that contains a form, and then let users fill out that form and e-sign it and collect the results, and then print them out all at once, it would be great. It's not a bil…
I assume you mean a drawn form as opposed to a true PDF form. The former would be difficult to parse automatically into inputs.
OTOH, a PDF form works exactly they way you’d like. Maybe there’s a small market in helping convert one to the other for collecting input from old paper-ish forms.
Re: What's so hard about PDF text extraction?
#16I suppose the best approach is to combine OCR techniques while taking hints from the PDF structure.
An order of magnitude increase of time is very significant. If you're just processing a few documents with a lot of human oversight you may be right, but it's definitely not a generalised best approach, at least going by the article.
Re: What's so hard about PDF text extraction?
#17On a personal project, I had a good experience extracting PDF text using Tabula[1]. You specify the bounding boxes where desired data is, and it spits out the content it finds. It still hits the issues mentioned in this article (surprise spaces appearing in middle of words, etc) [1] https://tabula.technology/
There's also camelot in Python [1]. Discovered it on HN [2]. Still a decent amount of manual work afterwards though but it's probably unreasonable to expect otherwise.
Re: What's so hard about PDF text extraction?
#18The best technique for having a PDF with extractable data is to include the data within the PDF itself. That is what LibreOffice can do, it can slip in the entire original document within a PDF. Since a compressed file is quite small, the resulting files are not that much larger, and then you don't need to fuss with OCR or anything else.
As long as you can trust that the contents of the embedded document is the same as what is displayed.
Re: What's so hard about PDF text extraction?
#19The relatively small company I work for makes me fill out some forms by hand, because they receive them from vendors as a PDF. So I print it out, sign it, and return it to my company by hand. If someone could make a service that lets you upload a PDF that contains a form, and then let users fill out that form and e-sign it and collect the results, and then print them out all at once, it would be great. It's not a bil…
I use Xournal - https://sourceforge.net/projects/xournal/
It lets me type in to forms - or draw text over them if necessary. Then I paste in a scan of my signature. Then save as a PDF an email across.
I've been doing this for years. Job applications, mortgages, medical questionnaires. No one has every queried it.
If you're hand delivering a printed PDF, it's just going to be copy-typed by a human into a computer. No need to make it too fancy.
Re: What's so hard about PDF text extraction?
#20So all through this I’m thinking “just OCR it and be done”, and we get to:
> Why not OCR all the time? > Running OCR on a PDF scan usually takes at least an order of magnitude longer than extracting the text directly from the PDF.
... so? Google can OCR video and translate it in something that feels like real-time; what PDF processing are they doing that is so performance bound?
> Difficulties with non-standard characters and glyphs OCR algorithms have a hard time dealing with novel characters, such as smiley faces, stars/circles/squares (used in bullet point lists), superscripts, complex mathematical symbols etc.
Sure, but more than the random shit you find in PDFs anyway?
> Extracting text from images offers no such hints
Finding an algorithm that approximates how a human approaches a page layout doesn’t feel like it would be all that hard.
Obviously it’s very easy to stand on the sidelines and throw stones, but parsing PDFs using anything other than OCR + some machine learning models to work out what the type of a piece of text feels like pretending we are still constrained by the processing costs of 5 years ago