The open-source Ghostscript [1] can convert simple PDFs to text, while keeping the layout. I doubt it will handle some of the more complicated cases outlined in the article though. I use it quite successfully to turn my bank statements into text, which can then be further processed. [1]: https://www.ghostscript.com/
MuPDF and XPDF can do that by selecting the text with the right mouse buttom.
What's so hard about PDF text extraction?
151–160 of 350 posts
Re: What's so hard about PDF text extraction?
#152I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…
Any tricks for decimal points versus noise? Its a terrifying outcome and all I've got is doing statistical analysis on the data you've already got and highlighting "outliers".
Re: What's so hard about PDF text extraction?
#153Earlier quoted context omitted.
Maybe there's a SasS opportunity for you to explore.
Hi! I’m the founder of a startup ( https://siftrics.com ) in this exact space. The demand for automating text extraction is still very high — or at least it feels like it when you’re working around the clock to cater to 3 of your customers, only to wake up to 10 more the next day. We’re small but growing extremely quickly.
I've bookmarked your site for future research... but the aviation part has me curious!
Re: What's so hard about PDF text extraction?
#154If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…
Right? I love the OCR for Google Drive. It's such a useful, hidden feature. By the way, why do you wait 10 minutes? Is there a signal that the PDF is done processing? Or is there just some kind of voodoo magic that seems to happen that just takes 10 minutes to do?
Also, I am not aware of a signal when it is done.
Re: What's so hard about PDF text extraction?
#155I walked away from the product over a decade ago since it always seemed like it’d be trivial for adobe to implement the feature in reader. Though every couple of years there’s a redaction scandal and I keep wondering how lucrative the product could have been with some marketing.
Re: What's so hard about PDF text extraction?
#156How you process PDF depends greatly on the scale at which you're working with documents. For large-volume, high-speed processing, automation is necessary. Where you're translating a more stable corpus, human input may be tractable. The ability to look at source PDF, OCR, and an edited text version to correct for errors seems a part of that workflow.
Often it's possible to get close or approximate transcription using standard tools. I've found the Poppler library's "pdftotext" remarkably good with many PDFs, so long as there's some text within them: https://poppler.freedesktop.org
There's a general concept I've been working toward of a minimum sufficient document complexity, which follows a rough (though not strict) hierarchy. It's remarkable how much online content is little more than paragraph-separated text, with no further structure. Even images are not strictly informational, but rather window-dressing.
Typically, additional elements added are hyperlinks, images, text emphasis (italic and bold, often only the first), sections, lists, blockquotes, super- and sub-script, in roughly that order.
(A study looking at the prevalence of specific semantic HTML elements within a corpus would be ... interesting.)
Then there are the elements NOT natively supported in HTML: equations, endnotes/footnotes, tables of contents, etc.
It seems to me there should be an analogue to Komolgrov complexity as concerns layout of textual documents. That is: there is a minimum necessary and sufficient level of markup (perhaps: number, type, and relationship of elements) necessary to lay out a specific work.
I've tagged out novel-length books in Markdown with little more than the occasional italic and chapter marks.
Documents which use more markup than is required are overspecified. This is the underlying problem with a great deal of layout, and the ability to reduce texts to their minimum complexity would be useful. It's a nontrivial problem, though large swathes of it should be reasonably achievable.
Another approach would be for information-exchange formats to actually be, you know, information exchange formats rather than PDF.
(Though the latter is often, though not always, well-suited to reading.)
Re: What's so hard about PDF text extraction?
#157PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.
We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…
Re: What's so hard about PDF text extraction?
#158Earlier quoted context omitted.
I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down. The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing. Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice. Off course there were workarounds…
I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.
Re: What's so hard about PDF text extraction?
#159If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…
Now I think about it, I don't know what you mean by "upload a pdf to google drive and download it 10 minutes later".
Uploading and downloading a file shouldn't change it at all, at bit level.
Re: What's so hard about PDF text extraction?
#160I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…