A Python Library to extract tabular data from PDFs • Hacker News Board

Re: A Python Library to extract tabular data from PDFs

#31

Post by TeMPOraL » Fri, Oct 12, 2018, 12:18 PM UTC

aasasd wrote: post #22

aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

Sounds like OCR is really the universal method. I'm guessing it shouldn't even be as hard as full-blown OCR, since you have access to fonts used, so you can render known characters as a reference and pretty much run a per-pixel matching on rendered PDF.

Re: A Python Library to extract tabular data from PDFs

#32

Post by johnyesberg » Fri, Oct 12, 2018, 12:23 PM UTC

https://github.com/invoice-x/invoice2data is another one.

Re: A Python Library to extract tabular data from PDFs

#33

Post by decasteve » Fri, Oct 12, 2018, 12:24 PM UTC

This is nice. I do quite a bit of tabular data extraction and pdf tables are often a sticking point. It is absolutely correct in describing it as a "fuzzy" problem.

My go-to solution has been 'pdftotext -layout' with a bit of hackery before giving it to pandas.read_fwf. That usually gets me 80% of the way there 80% of the time. The upside is that this tends to fail "better" than some other options.

I look forward to kicking-the-tires with this on my test cases.

Re: A Python Library to extract tabular data from PDFs

#34

Post by plaidfuji » Fri, Oct 12, 2018, 1:01 PM UTC

> However, OpenCV’s Hough Line Transform returned only line equations.

Did you try HoughLinesP? https://docs.opencv.org/2.4/modules/imgproc/doc/feature_dete...

Returns line segment endpoints with a probabilistic Hough Transform. I'm fully confident your solution works, just wondering if you tried this and why it was rejected.

Re: A Python Library to extract tabular data from PDFs

#35

Post by vortex_ape » Fri, Oct 12, 2018, 1:10 PM UTC

plaidfuji wrote: post #34

> However, OpenCV’s Hough Line Transform returned only line equations. Did you try HoughLinesP? https://docs.opencv.org/2.4/modules/imgproc/doc/feature_dete... Returns line segment endpoints with a probabilistic Hough Transform. I'm fully confident your solution works, just wondering if you tried this and why it was rejected.

Hi plaidfuji! I did try HoughLinesP during experimentation. I vaguely remember (since this was almost 2 years back) getting the actual line segment as a combination of multiple smaller line segments in all cases (which could then be combined to form the actual segment using some heuristic). It came down to getting the actual table line segment out which a combination morphological transformations and cv2.findContours provided (without the need for another combining step).

Re: A Python Library to extract tabular data from PDFs

#36

Post by vortex_ape » Fri, Oct 12, 2018, 1:19 PM UTC

decasteve wrote: post #33

This is nice. I do quite a bit of tabular data extraction and pdf tables are often a sticking point. It is absolutely correct in describing it as a "fuzzy" problem. My go-to solution has been 'pdftotext -layout' with a bit of hackery before giving it to pandas.read_fwf. That usually gets me 80% of the way there 80% of the time. The upside is that this tends to fail "better" than some other options. I look forward to…

Do submit bugs on GitHub if you face any issues! https://github.com/socialcopsdev/camelot

Re: A Python Library to extract tabular data from PDFs

#37

Post by plaidfuji » Fri, Oct 12, 2018, 1:21 PM UTC

vortex_ape wrote: post #35

plaidfuji wrote: post #34

> However, OpenCV’s Hough Line Transform returned only line equations. Did you try HoughLinesP? https://docs.opencv.org/2.4/modules/imgproc/doc/feature_dete... Returns line segment endpoints with a probabilistic Hough Transform. I'm fully confident your solution works, just wondering if you tried this and why it was rejected.

Hi plaidfuji! I did try HoughLinesP during experimentation. I vaguely remember (since this was almost 2 years back) getting the actual line segment as a combination of multiple smaller line segments in all cases (which could then be combined to form the actual segment using some heuristic). It came down to getting the actual table line segment out which a combination morphological transformations and cv2.findContours…

Interesting. I noticed you mentioned below that you're trying to get rid of OpenCV as a dependency - that's really tough. I came from a Matlab background where image processing was really well-packaged and Python is a total mess.

If you managed to vendor a small portion of OpenCV that contained image i/o, basic colorspace conversion, thresholding, scaling/rotating, shape drawing/insertion, HoughLines and findContours, I think you could release that as its own package and it would be quite popular. OpenCV is such a bloated dependency...

Re: A Python Library to extract tabular data from PDFs

#38

Post by vortex_ape » Fri, Oct 12, 2018, 1:25 PM UTC

KhalilK wrote: post #30

A few months ago I was looking for a similar solution but couldn't find one that handles empty cells very well. I ended up writing my own program[0] that is specific to my files' layout. This library works perfectly and could've saved me a lot of time! Looking at some of the source code, we used similar logic to parse the tables. Pretty neat! 0. https://github.com/khllkcm/pdf2calendar

Will check out pdf2calendar!

Re: A Python Library to extract tabular data from PDFs

#39

Post by nerdponx » Fri, Oct 12, 2018, 1:26 PM UTC

Interesting. I've used Tabula [0] in the past with great success. I wonder how this compares.

[0]: https://github.com/tabulapdf/tabula

Re: A Python Library to extract tabular data from PDFs

#40

Post by radarsat1 » Fri, Oct 12, 2018, 1:28 PM UTC

> When using Stream, tables aren’t autodetected. Stream treats the whole page as a single table

I've often wondered if image semantic segmentation methods as used in the ML community could successfully identify things like "there is a table (or figure) here, it's not part of the main text". I mean, it seems that humans should be able to do this even without reading the text so I don't see why a CNN couldn't.