Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

71–80 of 350 posts

Re: What's so hard about PDF text extraction?

#71

Post by penagwin »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Id also be interested in a blog or any basic tips/examples! I totally understand you don't want to give too much away, but I'm sure HN would love to see it!

Re: What's so hard about PDF text extraction?

#72

Post by wenc »

dwheeler wrote: post #5

The best technique for having a PDF with extractable data is to include the data within the PDF itself. That is what LibreOffice can do, it can slip in the entire original document within a PDF. Since a compressed file is quite small, the resulting files are not that much larger, and then you don't need to fuss with OCR or anything else.

Yes to embedding. In Canada, folks have always been able to e-file tax returns, but the CRA (Canada Revenue Agency) also has fillable PDF form for folks who insist on mailing in their returns (with their receipts and stuff so they don't have to store them and risk losing them).

When you're done filling the form, the PDF runs form validity checks and generates a 2D barcode [1] -- which stores your all field entry data -- on the first page. This 2D barcode can then be digitally extracted on the receiving end with either a 2D barcode scanner or a computer algorithm. No loss of fidelity.

Looks like Acrobat supports generation of QR, PDF417 and Data Matrix 2D barcodes.[2]

[1] https://www.canada.ca/en/revenue-agency/services/tax/busines...

[2] https://helpx.adobe.com/acrobat/using/pdf-barcode-form-field...

Re: What's so hard about PDF text extraction?

#73

Post by siftrics »

slig wrote: post #34
daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Maybe there's a SasS opportunity for you to explore.

Hi! I’m the founder of a startup (https://siftrics.com) in this exact space.

The demand for automating text extraction is still very high — or at least it feels like it when you’re working around the clock to cater to 3 of your customers, only to wake up to 10 more the next day. We’re small but growing extremely quickly.

Re: What's so hard about PDF text extraction?

#74

Post by izacus »

jatsign wrote: post #13
travisporter wrote: post #10

Earlier quoted context omitted.

Would DocuSign work? I’ve signed for lease documents several times that way.

Something like that would work for signing, but the hard part is "turn this pdf into an online form". That way after a user finishes a form, you can perform some basic error checking like, did they fill out everything, is this field a valid format, etc. After 100 employees turn in a multi-page printed out form, someone has to go through it and make sure they signed everywhere, filled out all the fields, etc. Again, n…

Well, PDF forms support all of this already, so why not just add validation inside the PDF?

Re: What's so hard about PDF text extraction?

#75

Post by UglyToad »

The open source project I work on [0] returns the letters, their positions and other associated information.

We provide support for retrieving words as well as a bunch of different algorithms for document layout analysis [1]. But like the other commenters here mention, it's an extremely difficult problem which doesn't have an easy or general solution.

I was trying to build a custom library on top of the open-source library that did a bit more processing, multi-column analysis, statistical analysis of whitespace size, etc. But building something that works for the general case is difficult enough to be functionally impossible.

Despite that I think the PDF format is well suited to what it is for and there are very few "implementation mistakes" in the spec itself (no up-front length for inline image data is the main one, plus accessibility obviously). It's ultimately become too successful and as a result developers are stuck handling cases where it's being used for entirely the wrong purpose but I can't see a way to another format gaining purchase for the correct purpose (perhaps it's like JavaScript in that way, it has huge adoption because it was first, not because it does all jobs well).

Perhaps a content-first format which also handles presentation well could gain a foothold if it came with a shim for PDF viewers and software to use but I dread to think how much effort that would be.

[0]:https://github.com/UglyToad/PdfPig

[1]:https://github.com/UglyToad/PdfPig/wiki/Document-Layout-Anal...

Re: What's so hard about PDF text extraction?

#76

Post by bhanhfo »

On the other hand... OCR is meanwhile so good that it can be used for many PDF text extraction projects. So often there is no longer the need to bother with PDF internals, just screenshot the PDF document and parse it. A free pdf ocr service is for example ocr.space.

Re: What's so hard about PDF text extraction?

#77

Post by gruez »

wenc wrote: post #72
dwheeler wrote: post #5

The best technique for having a PDF with extractable data is to include the data within the PDF itself. That is what LibreOffice can do, it can slip in the entire original document within a PDF. Since a compressed file is quite small, the resulting files are not that much larger, and then you don't need to fuss with OCR or anything else.

Yes to embedding. In Canada, folks have always been able to e-file tax returns, but the CRA (Canada Revenue Agency) also has fillable PDF form for folks who insist on mailing in their returns (with their receipts and stuff so they don't have to store them and risk losing them). When you're done filling the form, the PDF runs form validity checks and generates a 2D barcode [1] -- which stores your all field entry data…

>for folks who insist on mailing in their returns (with their receipts and stuff so they don't have to store them and risk losing them).

The Canadian tax agency offers free storage for whatever receipts you mail them? Sounds nifty. Does the IRS (or any other tax agency) do this?

Re: What's so hard about PDF text extraction?

#78

Post by pacoverdi »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Many years ago, I regularly had to parse specifications of protocols from various electronic exchanges. The general approach I used was to do a first pass using a Linux tool to convert it to text: pdftotext. Something like:

    pdftotext -layout -nopgbrk -eol unix -f $firstpage -l $lastpage -y 58 -x 0 -H 741 -W 596 "$FILE"
After that, it was a matter of writing and tweaking custom text parsers (in python or java) until the output was acceptable, generally an XML file consumed by the build (mainly to generate code).

A frequent need was to parse tables describing fields (name, id, description, possible values etc.). Unfortunately, sometimes tables spanned several pages and the column width was different on every page, which made column splitting difficult. So I annotated page jumps with markers (e.g. some 'X' characters indicating where to cut).

As someone else said, this is like black magic, but kind of fun :)

Edit: grammar

Re: What's so hard about PDF text extraction?

#79

Post by saradhi »

I'm an ML engineer, worked as a part time data engineer consultant for a medical lines/claims extraction company, for 3 years, which majorly involved in extracting the tabular data from the PDFs and Images. Developer rules or parsers as such is JUST no help. You end up creating a new rule every time you miss the data extraction.

With that in consideration, and the existing resources are little help especially on skewed, blurry, handwritten and 2 different table structure in the input, I ended up creating an API service to extract tabular data from Images and PDFs - hosted as https://extracttable.com . We cared it to be robust, average extraction time on images is under 5 seconds. On top of maintaining accuracy, A bad extraction is eligible for credit usage refund, which literally not any service offer it.

i Invite HN users to give it a try and feel free to email saradhi@extracttable.com for extra API credits for the trail.

Re: What's so hard about PDF text extraction?

#80

Post by Iwillgetby »

If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same.

I have used Google's fine OCR results to simulate a hacker.

- Download a youtube video that shows how to attack a server on the website hackthebox.eu

- Run ffmpeg to convert the video to images.

- Run a jpeg to pdf tool.

- Upload the pdf to google drive.

- Download the pdf from google drive.

- Grep for the command line identifiers "$" "#".

- Connect to hackthebox.eu vpn.

- Attack the same machine in the video.

Post reply on HN