Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

Post reply on HN
Print view View original

41–50 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#41

Post by aidos »

vortex_ape wrote: post #18
aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

The extreme cases (where there is no/incorrect mapping between the glyph and the character they represent) are a real pain! This mapping is stored as a ToUnicode map inside the PDF. In the past I've used OCR to handle such cases but I'm planning to create an experimental interface where anyone can modify the ToUnicode map. The challenge would be to make the modifications automated/user friendly.

Wicked. Sometimes the ToUnicode map is missing so we actually rebuild it ourselves from other information we find in the PDF.

Re: A Python Library to extract tabular data from PDFs

#42

Post by aidos »

aasasd wrote: post #22
aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

I think it's probably Stockholm syndrome but I quite enjoying digging into PDFs. There are actually other ways of embedding the text where you have more information to go on. "Characters at coordinates" is a particularly rough hand to be dealt in a PDF (though it definitely happens).

Re: A Python Library to extract tabular data from PDFs

#43

Post by nabla9 »

aasasd wrote: post #29
spaceandshit wrote: post #26

Earlier quoted context omitted.

And replaced with what?

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

HTML and PDF are used for different purposes.

PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that.

HTML is markup for presenting documents application dependent manner.

Re: A Python Library to extract tabular data from PDFs

#44

Post by chedar »

nerdponx wrote: post #39

Interesting. I've used Tabula [0] in the past with great success. I wonder how this compares. [0]: https://github.com/tabulapdf/tabula

They have a detailed comparison with other tools (including Tabula) in the wiki:

https://github.com/socialcopsdev/camelot/wiki/Comparison-wit...

Re: A Python Library to extract tabular data from PDFs

#46

Post by aasasd »

nabla9 wrote: post #43
aasasd wrote: post #29

Earlier quoted context omitted.

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

HTML and PDF are used for different purposes. PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that. HTML is markup for presenting documents application dependent manner.

They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs.

I have dozens of PDFs in my reading queue, for which I'll probably have to buy a tablet. Why can't I read the same columns of text and pictures on my electroink reader when I can do that with HTML? Who the hell knows.

Re: A Python Library to extract tabular data from PDFs

#47

Post by svat »

aasasd wrote: post #22
aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

The great thing about PDF, even the reason for its existence and adoption, is that a (valid) PDF file will look exactly the same — the same characters in the same fonts at exactly the same positions on every page — on any printer or display across the world, and across time. With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers.

Sometimes that's what you want (and when the visual appearance is not important, it may make sense to not use PDF), but I definitely wouldn't want to see PDF “dead”.

Re: A Python Library to extract tabular data from PDFs

#48

Post by nabla9 »

aasasd wrote: post #46
nabla9 wrote: post #43

Earlier quoted context omitted.

HTML and PDF are used for different purposes. PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that. HTML is markup for presenting documents application dependent manner.

They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…

PDF is for long term archival.

There is no standard and widely recognized long term archival format for HTML pages (with all the extras). Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large.

You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archieve files you start to see how they become crap over time.

Post reply on HN