Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

11–20 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#11
post #9

Wouldn't it be easier and more generic to have an OCR solution for this task?

Hey amelius! Though OCR would provide a generic solution, it would be an overkill for text-based PDFs. I'm working on getting a OCR solution up since there's still a lot of data that is trapped inside scanned PDFs and not text-based ones.

If you have any pointers in the OCR route, do suggest them here, or on this GitHub issue! https://github.com/socialcopsdev/camelot/issues/101

Re: A Python Library to extract tabular data from PDFs

#12
post #7

Earlier quoted context omitted.

Try these: Page 233 http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8351-M... Page 45 https://ae-bst.resource.bosch.com/media/_tech/media/datashee... Is the library able to handle cells that span multiple columns?

I assumed that you're talking about page 33 in the first PDF, since it has only 225 pages. I extracted Figure 6-23 from it and the table on page 45 in the second PDF. Here's a gist: https://gist.github.com/vinayak-mehta/cf30a5560f1b8ab4c0b25e... Yes, Camelot takes care of cells spanning multiple columns! You can check out the Advanced Usage section for explanation on the keyword arguments I used in the gist! https://…

Note: I had to decrypt the second PDF using qpdf since the library I'm using to split a PDF into pages (PyPDF2) doesn't support the encryption type of that PDF.

Did this: qpdf --decrypt input.pdf output.pdf

Re: A Python Library to extract tabular data from PDFs

#13

This is a very interesting software. In research community still many results are only in pdf tables in papers, so obtaining them in dataframe is very useful, good job!. By the way, I would like export also in Excel files in the command line.

Hey danimolina! You can export the data into an excel by specifying it as the export format, Camelot comes with a command-line interface too! https://camelot-py.readthedocs.io/en/master/user/cli.html#cl...

You can simple do: camelot --output data.xlsx --format excel lattice input.pdf (lattice can be replaced with stream based on the type of tables in your PDF)

Re: A Python Library to extract tabular data from PDFs

#14
Quick suggestion - you should integrate the functions to extract signature data inside PDF. This is a huge issue and everyone has to write their own.

for example, this is my sample piece of code to extract data from Aadhaar signed PDF https://pastebin.com/dg8p98T1

Re: A Python Library to extract tabular data from PDFs

#15

Quick suggestion - you should integrate the functions to extract signature data inside PDF. This is a huge issue and everyone has to write their own. for example, this is my sample piece of code to extract data from Aadhaar signed PDF https://pastebin.com/dg8p98T1

Thanks for the suggestion sandGorgon! Can you also point me to an example of a PDF with signature data?

Re: A Python Library to extract tabular data from PDFs

#16

Quick suggestion - you should integrate the functions to extract signature data inside PDF. This is a huge issue and everyone has to write their own. for example, this is my sample piece of code to extract data from Aadhaar signed PDF https://pastebin.com/dg8p98T1

Thanks for the suggestion sandGorgon! Can you also point me to an example of a PDF with signature data?

unfortunately i cannot share without running afoul of all the laws out there. but you can create your own here - https://app.digio.in/#/authenticate

Re: A Python Library to extract tabular data from PDFs

#18
post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

The extreme cases (where there is no/incorrect mapping between the glyph and the character they represent) are a real pain! This mapping is stored as a ToUnicode map inside the PDF. In the past I've used OCR to handle such cases but I'm planning to create an experimental interface where anyone can modify the ToUnicode map. The challenge would be to make the modifications automated/user friendly.

Re: A Python Library to extract tabular data from PDFs

#19

Earlier quoted context omitted.

Thanks for the suggestion sandGorgon! Can you also point me to an example of a PDF with signature data?

unfortunately i cannot share without running afoul of all the laws out there. but you can create your own here - https://app.digio.in/#/authenticate

Ah sorry I forgot about posting PII data online. Thanks for the link!
Post reply on HN