Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

151–160 of 350 posts

Re: What's so hard about PDF text extraction?

#151
post #86
post #82

The open-source Ghostscript [1] can convert simple PDFs to text, while keeping the layout. I doubt it will handle some of the more complicated cases outlined in the article though. I use it quite successfully to turn my bank statements into text, which can then be further processed. [1]: https://www.ghostscript.com/

MuPDF and XPDF can do that by selecting the text with the right mouse buttom.

xpdf seems to have started to respect the not-copyable-flags, while in days yonder, it didn't. So now, even something like a manual of some command-line tools or a text book on C++ or Rust, you still have to re-type the text (wtf). Time to remove and search for something better, something that does not need a 0.5GB update every 3 days (on Windows). (yes, exaggerating slightly)

Re: What's so hard about PDF text extraction?

#152

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Any tricks for decimal points versus noise? Its a terrifying outcome and all I've got is doing statistical analysis on the data you've already got and highlighting "outliers".

I've done this a bit. I define ranges per numeric field and if it exceeds or is below that range, I send it to another queue for manual review. Sometimes I'll write rules where if it's a dollar amount that usually ends ".00" and I don't read a decimal but I do have "00", then I'll just fix that automatically if it's outside my range.

Re: What's so hard about PDF text extraction?

#153
post #34

Earlier quoted context omitted.

Maybe there's a SasS opportunity for you to explore.

Hi! I’m the founder of a startup ( https://siftrics.com ) in this exact space. The demand for automating text extraction is still very high — or at least it feels like it when you’re working around the clock to cater to 3 of your customers, only to wake up to 10 more the next day. We’re small but growing extremely quickly.

As someone who works in aviation... what made you choose an avionics company as your demo business?

I've bookmarked your site for future research... but the aviation part has me curious!

Re: What's so hard about PDF text extraction?

#154

If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…

Right? I love the OCR for Google Drive. It's such a useful, hidden feature. By the way, why do you wait 10 minutes? Is there a signal that the PDF is done processing? Or is there just some kind of voodoo magic that seems to happen that just takes 10 minutes to do?

2 minutes is probably long enough. I did notice that google drive doesn't seem to like it if you upload a lot of files. I have had files sit and never get OCR, but I forgot about them so they may have OCR on them now.

Also, I am not aware of a signal when it is done.

Re: What's so hard about PDF text extraction?

#155
FYI, redaction from pdf can be similarly difficult. I once was tangentially involved with a pdf redaction piece of software, and due to many different issues with pdf, the solution ended up being to create images of the input pdf draw over the redacted info, then create/overwrite a new pdf that was just a container for the jpgs. It was the only way to be sure the info wasn’t in the pdf at all, since it could be all kinds of places and duplicated in interesting ways. But since you’d be working on the final rendering you could be sure everything you covered would be covered in the final output. The biggest challenges after that were related to text extraction, since we wanted a nice UI where you could select text and the redaction would auto cover text and use a uniform width based on the heights of all characters in the redaction. I think more often than we were happy with a user would need to simply use a bounding box since extracting all the pertinent data related to the text was so hard.

I walked away from the product over a decade ago since it always seemed like it’d be trivial for adobe to implement the feature in reader. Though every couple of years there’s a redaction scandal and I keep wondering how lucrative the product could have been with some marketing.

Re: What's so hard about PDF text extraction?

#156
I've been wrestling with a similar set of tasks, and have arrived at a similar set of tools and options.

How you process PDF depends greatly on the scale at which you're working with documents. For large-volume, high-speed processing, automation is necessary. Where you're translating a more stable corpus, human input may be tractable. The ability to look at source PDF, OCR, and an edited text version to correct for errors seems a part of that workflow.

Often it's possible to get close or approximate transcription using standard tools. I've found the Poppler library's "pdftotext" remarkably good with many PDFs, so long as there's some text within them: https://poppler.freedesktop.org

There's a general concept I've been working toward of a minimum sufficient document complexity, which follows a rough (though not strict) hierarchy. It's remarkable how much online content is little more than paragraph-separated text, with no further structure. Even images are not strictly informational, but rather window-dressing.

Typically, additional elements added are hyperlinks, images, text emphasis (italic and bold, often only the first), sections, lists, blockquotes, super- and sub-script, in roughly that order.

(A study looking at the prevalence of specific semantic HTML elements within a corpus would be ... interesting.)

Then there are the elements NOT natively supported in HTML: equations, endnotes/footnotes, tables of contents, etc.

It seems to me there should be an analogue to Komolgrov complexity as concerns layout of textual documents. That is: there is a minimum necessary and sufficient level of markup (perhaps: number, type, and relationship of elements) necessary to lay out a specific work.

I've tagged out novel-length books in Markdown with little more than the occasional italic and chapter marks.

Documents which use more markup than is required are overspecified. This is the underlying problem with a great deal of layout, and the ability to reduce texts to their minimum complexity would be useful. It's a nontrivial problem, though large swathes of it should be reasonably achievable.

Another approach would be for information-exchange formats to actually be, you know, information exchange formats rather than PDF.

(Though the latter is often, though not always, well-suited to reading.)

Re: What's so hard about PDF text extraction?

#157
post #69

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…

Any recommended library for .NET to extract text by coordinates?

Re: What's so hard about PDF text extraction?

#158
post #129
post #60

Earlier quoted context omitted.

I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down. The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing. Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice. Off course there were workarounds…

I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.

I blame CSS.

Re: What's so hard about PDF text extraction?

#159

If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…

Google Drive can directly OCR jpeg or any image. Just upload and open it with Google Docs.

Now I think about it, I don't know what you mean by "upload a pdf to google drive and download it 10 minutes later".

Uploading and downloading a file shouldn't change it at all, at bit level.

Re: What's so hard about PDF text extraction?

#160

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

What’s your contact info? Didn’t see any on your github.
Post reply on HN