A Python Library to extract tabular data from PDFs • Hacker News Board

Re: A Python Library to extract tabular data from PDFs

#21

Post by squaresmile » Fri, Oct 12, 2018, 11:01 AM UTC

This sounds neat. Thanks for the work, vortex_ape and others. When I last needed this, I used tabula via tabula-py. Tried camelot on the PDF [0] I worked on and unfortunately the default option returned less-workable dataframe than tabula-py. I think it's just the area detection of stream and you are working on it anyway so I'm really looking forward to see the results.

btw, I think the pip install requirements missed opencv-python (on Windows?). And in this doc [2], it should be "top left and bottom right" instead of "left-top and right-bottom".

[1] https://www.boj.or.jp/en/statistics/set/kess/release/2018/ke...

[2] https://camelot-py.readthedocs.io/en/master/user/advanced.ht...

Re: A Python Library to extract tabular data from PDFs

#22

Post by aasasd » Fri, Oct 12, 2018, 11:01 AM UTC

aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package.

I so much want to see the day when PDF is dead like Flash.

Re: A Python Library to extract tabular data from PDFs

#23

Post by jacquesm » Fri, Oct 12, 2018, 11:02 AM UTC

Oh that is so timely. I've been putting that part of a pipeline I built off for a while due to the complexity and now I can just plug this in. Super neat. Thank you very much!

Re: A Python Library to extract tabular data from PDFs

#24

Post by vortex_ape » Fri, Oct 12, 2018, 11:09 AM UTC

squaresmile wrote: post #21

This sounds neat. Thanks for the work, vortex_ape and others. When I last needed this, I used tabula via tabula-py. Tried camelot on the PDF [0] I worked on and unfortunately the default option returned less-workable dataframe than tabula-py. I think it's just the area detection of stream and you are working on it anyway so I'm really looking forward to see the results. btw, I think the pip install requirements misse…

Hey squaresmile! Yes, right now table detection with Stream doesn't work nicely if the table is not present on the full page, for which you can use the table_area kwarg from [2].

You should use "pip install camelot-py[all]" to install Camelot (which will install opencv-python too). I had to take it out of the requirements since it wasn't available in any conda channels while I was creating the conda package. I'm looking to remove opencv as a requirement altogether by either vendorizing the opencv code that is being used inside Camelot or reimplementing the code using something lightweight like pillow.

Thanks for the catch in [2], I'll correct it!

Re: A Python Library to extract tabular data from PDFs

#25

Post by vortex_ape » Fri, Oct 12, 2018, 11:23 AM UTC

jacquesm wrote: post #23

Oh that is so timely. I've been putting that part of a pipeline I built off for a while due to the complexity and now I can just plug this in. Super neat. Thank you very much!

What does this pipeline do and what software have you used to implement it?

I have used Airflow in the past to create ETL pipelines, and plugged in Camelot in one of them to extract tables from PDFs. I also wrote a blog post about it in case you might be interested. https://hackernoon.com/how-to-create-a-workflow-in-apache-ai...

Re: A Python Library to extract tabular data from PDFs

#26

Post by spaceandshit » Fri, Oct 12, 2018, 11:31 AM UTC

aasasd wrote: post #22

aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

And replaced with what?

Re: A Python Library to extract tabular data from PDFs

#27

Post by vortex_ape » Fri, Oct 12, 2018, 11:37 AM UTC

aasasd wrote: post #22

aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

Adobe released a way to attach data tables with PDFs. But I think it hasn't been adopted fully, since many organizations that release open data as PDFs don't tag the accompanying data tables. https://www.w3.org/TR/WCAG20-TECHS/PDF6.html

Re: A Python Library to extract tabular data from PDFs

#28

Post by aasasd » Fri, Oct 12, 2018, 11:51 AM UTC

amelius wrote: post #9

Wouldn't it be easier and more generic to have an OCR solution for this task?

It seems that generally you'd want all functions of an OCR engine aside from the character recognition itself—namely layout detection. (And sometimes you'll need the character recognition too.)

I'd bet that commercial OCR packages that are long in the game have unified code for these functions between regular OCR and PDF processing.

Re: A Python Library to extract tabular data from PDFs

#29

Post by aasasd » Fri, Oct 12, 2018, 11:58 AM UTC

spaceandshit wrote: post #26

aasasd wrote: post #22

Earlier quoted context omitted.

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

And replaced with what?

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

Re: A Python Library to extract tabular data from PDFs

#30

Post by KhalilK » Fri, Oct 12, 2018, 12:07 PM UTC

A few months ago I was looking for a similar solution but couldn't find one that handles empty cells very well. I ended up writing my own program[0] that is specific to my files' layout.

This library works perfectly and could've saved me a lot of time! Looking at some of the source code, we used similar logic to parse the tables. Pretty neat!

0.https://github.com/khllkcm/pdf2calendar