Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#61

Post by aasasd »

danso wrote: post #54
aasasd wrote: post #46

Earlier quoted context omitted.

They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…

PDF is for printing and for (well, with the exception of a few edge cases) guaranteeing a layout and display given a set of paper dimensions. HTML has the advantage of responsiveness, but the inherent problem of variable output. When I was a professor and advising my students on creating portfolios, I told them to build websites of course. But I told them to also have a link to a one-page PDF because many organizatio…

Variable output is not a problem, it's exactly what's needed. The days of standard paper formats are over, old man: in a decade people will have documents delivered straight to their retinas, or read into their ears—but everyone will still have to scroll PDFs back and forth with no possibility of reformatting, because lots of papers are published in it.

If you want to have your document printed nicely, just prepare it for printing along with other methods of output. The best way to do it is to not use some crazy layout: have a single column with images between paragraphs, and your documents will look fine on any device. All problems of reformatting documents stem from the rigid two-dimensional layout mentality, while the flexible approach requires stepping back to the one-dimensional semantic flow.

(Actually, standard paper formats were never around, because—surprise—my country doesn't use US paper formats.)

Re: A Python Library to extract tabular data from PDFs

#62

Post by pwg »

aasasd wrote: post #22
aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

> all functions of an OCR engine outside of character recognition per se

Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself.

PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a reverse mapping table when one does this that says "a decimal 32 byte prints a capitol letter Z" to allow for search and extraction purposes. But the PDF spec. does not require this reverse table be present.

So it is quite possible to randomly assign font glyphs to arbitrary byte values, and omit the reverse mapping table. The result would be that extracting data back out of that PDF results in garbage if one does not know beforehand what the mapping from byte value to glyph was.

So, if a 'bad actor' did this, one's only recourse to retrieving data would be to rasterize the PDF to a bitmap, then OCR the resulting bitmap to extract the content back out.

Re: A Python Library to extract tabular data from PDFs

#63

Post by bosie »

aasasd wrote: post #60
bosie wrote: post #58

Earlier quoted context omitted.

While I agree that PDFs are at times cumbersome to use, I can't think of a valid solution to replace them. - Fixed layout seems much easier to handle than dynamic layouts. I.e. I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long). And without handling this properly, most of the a…

> I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long) I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind. As far as I can tell, it's specific to when authors put images somewhere distant to the text that mentions them in t…

> I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind.

Not even trying to be funny but do you mind sharing some websites that dynamically resize content correctly? I just checked a couple of the usual suspects (reuters, nytimes, guardian, github) and none do it. They are all using (semi-)fixed layouts.

Re: A Python Library to extract tabular data from PDFs

#64

Post by aasasd »

nabla9 wrote: post #48
aasasd wrote: post #46

Earlier quoted context omitted.

They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…

PDF is for long term archival. There is no standard and widely recognized long term archival format for HTML pages (with all the extras). Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archieve files you start to see how the…

HTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format.

> Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large.

Not enough how? What is there that you need besides what the server hands to you, if that's what rendered in the first place? What magical compression methods do you have in PDF that are better than ZIP compression used in MAFF, for example?

> You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archive files you start to see how they become crap over time.

Have a static HTML version that's rendered the same in the future. You know, the same way that you have a static PDF standard.

How do you render Javascript in PDFs in a standard way? You don't use Javascript, that's how. Javascript is not for publication of static semantic text, so you don't use Javascript for papers, it's a no-brainer.

Re: A Python Library to extract tabular data from PDFs

#65

Post by tomrod »

pwg wrote: post #62
aasasd wrote: post #22

Earlier quoted context omitted.

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

> all functions of an OCR engine outside of character recognition per se Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself. PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a re…

I've had this issue with LaTeX produced docs before (scans bad actor).

Re: A Python Library to extract tabular data from PDFs

#66

Post by aasasd »

bosie wrote: post #63
aasasd wrote: post #60

Earlier quoted context omitted.

> I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long) I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind. As far as I can tell, it's specific to when authors put images somewhere distant to the text that mentions them in t…

> I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind. Not even trying to be funny but do you mind sharing some websites that dynamically resize content correctly? I just checked a couple of the usual suspects (reuters, nytimes, guardian, github) and none do it. They are all using (semi-)fixed layouts.

I have the opposite problem of finding a page that could be problematic. Flipped through several articles on those sites, and they all use the linear article layout.

Remember that we're talking about publication of static papers, so you look at the main content column on a page, since that's what should be there in a paper. In the main column, those sites use the simple linear flow: ‘text, image, text’—with images occupying entire paragraphs instead of floating to the sides. With this layout, you can reformat articles every which way, string them into horizontal pages, render them in columns or read them with text-to-speech, etc. It's essentially HTML 2.0 layout but with better formatting. Markdown readmes on Github are the perfect example of this approach.

I've regularly used Evernote for capturing web pages, and Pocket to read them on the phone, and they have no problem with storing main content from such articles, stripped of extraneous navigation (outside of Pocket's bugs with dropping some content, presumably from overzealous anti-ad measures).

You don't look at images outside of the main content column for this discussion, because those aren't what should be there in static paper-like publications—unless the images are related to the content. And if the images are related to the content, the question is why the author is trying to use a fancy layout for such a publication.

(NYTimes do sometimes use more complex layouts in feature articles, with dynamic effects—but they, presumably, don't target those for long-term archival, and instead they customize the pages for mobile and desktop access separately. Anyway, they also should tone that down if they want readership via something like Pocket.)

I most often have problems with images on Wikipedia, because they make images float to the right side since they have many non-essential but illustrative images. Those, indeed, tend to detach from the relevant text.

Re: A Python Library to extract tabular data from PDFs

#67

Post by aasasd »

svat wrote: post #47
aasasd wrote: post #22

Earlier quoted context omitted.

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

The great thing about PDF, even the reason for its existence and adoption, is that a (valid) PDF file will look exactly the same — the same characters in the same fonts at exactly the same positions on every page — on any printer or display across the world, and across time. With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices o…

Yeah, except the devices are different, so there's no point in trying to use the same page size on them, you only torture the reader.

> With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers.

Only if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal. Use a simple ‘text, image, text’ layout like it's the days of HTML 2.0, but with better formatting—and you'll have zero problems reformatting for different displays or reflowing the document into columns. Notice how all popular content sites adopted this layout in the main content column of their pages—and the pages work nicely on both desktop and mobile devices, and are captured fine with Pocket, Evernote and the like.

If you're trying to use a fancy layout for a paper-like publication, the question is why the hell you're doing that.

Re: A Python Library to extract tabular data from PDFs

#68

Post by nabla9 »

aasasd wrote: post #64
nabla9 wrote: post #48

Earlier quoted context omitted.

PDF is for long term archival. There is no standard and widely recognized long term archival format for HTML pages (with all the extras). Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archieve files you start to see how the…

HTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format. > Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. Not enough how? What is there that you need besides what the server hands to you, if that's what rendered in the first place? What magical compression methods do you have…

> HTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format.

HTML is not a good format and standard for that purpose. It's loose best effort markup with no good consensus on semantics. HTML with images is not good option for papers which have many equations.

EPUB3 is emerging standard for what you want, but it's not really good complete solution that can replace PDF/A or TeX/LaTeX

> Have a static HTML version that's rendered the same in the future

We don't have that.

Re: A Python Library to extract tabular data from PDFs

#69

Post by jononor »

plaidfuji wrote: post #37
vortex_ape wrote: post #35

Earlier quoted context omitted.

Hi plaidfuji! I did try HoughLinesP during experimentation. I vaguely remember (since this was almost 2 years back) getting the actual line segment as a combination of multiple smaller line segments in all cases (which could then be combined to form the actual segment using some heuristic). It came down to getting the actual table line segment out which a combination morphological transformations and cv2.findContours…

Interesting. I noticed you mentioned below that you're trying to get rid of OpenCV as a dependency - that's really tough. I came from a Matlab background where image processing was really well-packaged and Python is a total mess. If you managed to vendor a small portion of OpenCV that contained image i/o, basic colorspace conversion, thresholding, scaling/rotating, shape drawing/insertion, HoughLines and findContours…

scikit-image contains Hough transforms and the other things you mention? Though it does depends on scipy and matplotlib, which are kinda big.

Re: A Python Library to extract tabular data from PDFs

#70

Post by nuclx »

pwg wrote: post #62
aasasd wrote: post #22

Earlier quoted context omitted.

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

> all functions of an OCR engine outside of character recognition per se Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself. PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a re…

Or perform frequency analysis on the simple substitution chiffre. Seriously though, we need a document format with easier to extract payloads. Like Office documents with stronger structure, an underlying schema, along the lines of react-json-schema-form for Word.
Post reply on HN