Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

Post reply on HN
Print view View original

91–100 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#91

Post by danso »

xfer wrote: post #88
throwaway287391 wrote: post #85

Earlier quoted context omitted.

Okay, I could be convinced that a Markdown-type format with HTML output and very restrictive formatting could work, as long as it really includes (nearly) all of the support for tables, images, and math notation that LaTeX does, and displays well in any reasonable browser/platform. But is there an implemented working compiler of such a format? I sort of doubt it. (But if I'm wrong, please do post it.) Going back to R…

You should look at several papers written in scribble(racket doc templating language), they are beautiful, not hard to author(compared to latex). Here is an example: http://www.cs.utah.edu/plt/scope-sets . Although some of the models on semantics are pictures generated from latex :p

Wow, that indeed is a beautiful online book, better than any Sphinx template I've so far come across. You've got me interested in checking out Scribble.

Re: A Python Library to extract tabular data from PDFs

#92

Post by throwaway287391 »

xfer wrote: post #88
throwaway287391 wrote: post #85

Earlier quoted context omitted.

Okay, I could be convinced that a Markdown-type format with HTML output and very restrictive formatting could work, as long as it really includes (nearly) all of the support for tables, images, and math notation that LaTeX does, and displays well in any reasonable browser/platform. But is there an implemented working compiler of such a format? I sort of doubt it. (But if I'm wrong, please do post it.) Going back to R…

You should look at several papers written in scribble(racket doc templating language), they are beautiful, not hard to author(compared to latex). Here is an example: http://www.cs.utah.edu/plt/scope-sets . Although some of the models on semantics are pictures generated from latex :p

Thanks for the link -- this does look really nice. But I do think the fact that they resort to using LaTeX-generated images for text [1] (as you mentioned) sort of proves my point that the existing implementations of this sort of format aren't really flexible enough for a lot of scientific papers yet.

However, I admit this could be a reasonable way for even impatient and stubborn researchers to publish papers, given the right implementation. I'll withdraw my initial arrogant "I can tell you that ..." :P

[1] http://www.cs.utah.edu/plt/scope-sets/model.html#%28part._.S...

Re: A Python Library to extract tabular data from PDFs

#95

Post by aasasd »

xfer wrote: post #88
throwaway287391 wrote: post #85

Earlier quoted context omitted.

Okay, I could be convinced that a Markdown-type format with HTML output and very restrictive formatting could work, as long as it really includes (nearly) all of the support for tables, images, and math notation that LaTeX does, and displays well in any reasonable browser/platform. But is there an implemented working compiler of such a format? I sort of doubt it. (But if I'm wrong, please do post it.) Going back to R…

You should look at several papers written in scribble(racket doc templating language), they are beautiful, not hard to author(compared to latex). Here is an example: http://www.cs.utah.edu/plt/scope-sets . Although some of the models on semantics are pictures generated from latex :p

Scribble's styles might have been touched by Matthew Butterick, whose online-book “Practical Typography” may IMO be the most beautiful site on the whole web, owing to the masterful application of fonts and margins: https://practicaltypography.com/

(The list of MB's commits is not telling much, but Flatt having written the foreword to MB's other book “Beautiful Racket” is more suggestive.)

Re: A Python Library to extract tabular data from PDFs

#96

Post by emj »

nuclx wrote: post #70
pwg wrote: post #62

Earlier quoted context omitted.

> all functions of an OCR engine outside of character recognition per se Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself. PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a re…

Or perform frequency analysis on the simple substitution chiffre. Seriously though, we need a document format with easier to extract payloads. Like Office documents with stronger structure, an underlying schema, along the lines of react-json-schema-form for Word.

ODF - Open document format, it is not perfect but a lot better than Microsofts formats.

Re: A Python Library to extract tabular data from PDFs

#98

Post by kumartanmay »

vortex_ape wrote: post #11
amelius wrote: post #9

Wouldn't it be easier and more generic to have an OCR solution for this task?

Hey amelius! Though OCR would provide a generic solution, it would be an overkill for text-based PDFs. I'm working on getting a OCR solution up since there's still a lot of data that is trapped inside scanned PDFs and not text-based ones. If you have any pointers in the OCR route, do suggest them here, or on this GitHub issue! https://github.com/socialcopsdev/camelot/issues/101

Hey vortex_ape, we're also working on extracting data trapped inside scanned PDFs and recently, we've begun to get good results using DL algos. I am based in Gurugram, would you like to catch up and exchange experiences?

Re: A Python Library to extract tabular data from PDFs

#100

Post by kumartanmay »

squaresmile wrote: post #21

This sounds neat. Thanks for the work, vortex_ape and others. When I last needed this, I used tabula via tabula-py. Tried camelot on the PDF [0] I worked on and unfortunately the default option returned less-workable dataframe than tabula-py. I think it's just the area detection of stream and you are working on it anyway so I'm really looking forward to see the results. btw, I think the pip install requirements misse…

Are you also working on extracting tabular data from scanned image files?
Post reply on HN