Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#81

Post by maxxxxx »

aasasd wrote: post #22
aidos wrote: post #4

Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…

If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.

"I so much want to see the day when PDF is dead like Flash. "

Totally agree. A while ago I had to write code to import a ton of PDF files and it was just infuriating to realize that we have data in highly structured documents, throw all structure away to create a PDF and then we somehow have to divine that structure back from the PDF with enormous effort and only partial success. It's just a horrible, horrible file format for what it's used now.

Re: A Python Library to extract tabular data from PDFs

#82

Post by aidos »

aasasd wrote: post #29
spaceandshit wrote: post #26

Earlier quoted context omitted.

And replaced with what?

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

PDFs are used to build buildings from, for one. And there's a time when you want your layout to be exact :-)

Re: A Python Library to extract tabular data from PDFs

#83

Post by berti »

vortex_ape wrote: post #10
berti wrote: post #7

Earlier quoted context omitted.

Try these: Page 233 http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8351-M... Page 45 https://ae-bst.resource.bosch.com/media/_tech/media/datashee... Is the library able to handle cells that span multiple columns?

I assumed that you're talking about page 33 in the first PDF, since it has only 225 pages. I extracted Figure 6-23 from it and the table on page 45 in the second PDF. Here's a gist: https://gist.github.com/vinayak-mehta/cf30a5560f1b8ab4c0b25e... Yes, Camelot takes care of cells spanning multiple columns! You can check out the Advanced Usage section for explanation on the keyword arguments I used in the gist! https://…

Sorry! Page 203 in the first PDF has a full table if registers, and that bits in them. Thanks very much for the library, and even more for taking the time to create that notebook. The captured data looks excellent. I'll hax some code to translate this data to a header file suitable for writing a driver.

Re: A Python Library to extract tabular data from PDFs

#84

Post by bosie »

aasasd wrote: post #79
bosie wrote: post #77

Earlier quoted context omitted.

Sorry, i misunderstood you. I thought you wanted to move away from PDFs because they don't resize. But none of the example i gave resize either (neither does pocket or instapaper).

“Resize” is an ambiguous term, so you may or may not have understood me correctly, I'm still not sure. My (primary) problem with PDF is that it doesn't adapt to displays of different sizes—mobile, electroink and tablet devices in addition to desktop machines—and can't be reformatted on a device (e.g. to adjust the author's typesetting choices). Did you mean “zooming” the page in/out on the same device? That's not a b…

Fair point about font-resizing.

Since you are curious due to professional curiosity: what i meant by resize is the utilization of the device's screen. If my screen allows for a 1200px wide browser window, the main content shouldn't use 800px of it. On my 5000px wide screen, nytimes.com articles seem to utilize a whooping 10-15% (i am guessing). Might as well just send me a fixed-layout PDF.

That being said, I doubt it is computationally easy to compute a good layout. Considering how slow latex compiles a PDF, trying to find the optimal layout for a non-rigid layout seems difficult with the time constraint at hand.

Re: A Python Library to extract tabular data from PDFs

#85

Post by throwaway287391 »

aasasd wrote: post #80
throwaway287391 wrote: post #75

Earlier quoted context omitted.

As somebody who's written papers for scientific conferences and journals, I can tell you that this just isn't going to work. I'm not a designer and making a LaTeX-generated PDF with few frills look decent is enough work already. Generating HTML and having to verify that it looks good on every major browser whether it's a mobile or desktop device is just completely out of the question. Seriously, think about it -- ple…

For some reason I have to battle the same argument again and again in this thread. It's like people are blinded from having seen glamour magazines with crazy layouts. The solution is that you don't use fancy layouts for papers. Imagine that you only have HTML 2.0. Put your text in paragraphs, put images and large formulas in separate paragraphs between those of text. Now chuck that into a ready-made styling template…

Okay, I could be convinced that a Markdown-type format with HTML output and very restrictive formatting could work, as long as it really includes (nearly) all of the support for tables, images, and math notation that LaTeX does, and displays well in any reasonable browser/platform. But is there an implemented working compiler of such a format?

I sort of doubt it. (But if I'm wrong, please do post it.) Going back to Reddit comments again, even Reddit's version of Markdown (which only allows basic text formatting and simple tables, no images or math notation) is broken as hell in their own official mobile app (at least the iOS one). Tables are screwed up, and even bold/italic is somehow buggy. And there are (probably multiple) engineers paid full time SF salaries to work on Reddit's mobile app.

Also, there would still need to be a canonical print format for this to work with current conference/journal rules, which typically include page limits. And for good reason: nobody except the authors (least of all reviewers) want papers to be any longer than they are. (Sure, you could change to a word or character limit, but then you'd have unlimited images, which would incentivize stuffing tons of information and text into figures and using tons of those. So you'd have to bring in another requirement on total image size, or something. And you can see how this quickly gets overly complicated and you'd really rather just have a simple page limit.)

Re: A Python Library to extract tabular data from PDFs

#86

Post by svat »

aasasd wrote: post #67
svat wrote: post #47

Earlier quoted context omitted.

The great thing about PDF, even the reason for its existence and adoption, is that a (valid) PDF file will look exactly the same — the same characters in the same fonts at exactly the same positions on every page — on any printer or display across the world, and across time. With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices o…

Yeah, except the devices are different, so there's no point in trying to use the same page size on them, you only torture the reader. > With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers. Only if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal. Use a simple ‘te…

Yes, my idea of “the same” is “the same”.

I understand what I believe to be your actual point: it would be nice if documents were more often published in a format that doesn't completely fix their layout and visual appearance. And I agree with that! When I'm reading something purely for its information, and don't care too much about the appearance, I too would like it if it weren't in a visually-fixed format. (That's what I said in the first comment tot: “when the visual appearance is not important, […] not use PDF”.)

But my point is that for the goal of completely fixing the visual appearance, PDF is a pretty decent format (better than say, photographic images of the page), which is why it exists.

When you say you want “the day when PDF is dead”, it appears as though you cannot imagine anyone wanting that goal.

Here are two examples:

Suppose you are an author of books (a physical artefact that will inhabit libraries for centuries; forget about digital displays and all that nonsense for a moment) and care about their typographic quality. Then you will want to make sure of things like:

• that each paragraph contains appropriate line-breaks (http://eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf), so that the page as a whole has a good “texture” or “greyness” (or “colour”),

• that the words have hyphenation (to make the aforementioned good line-breaks possible), but not any poor hyphenation (https://tug.org/docs/liang/),

• that the typefaces chosen are in harmony with each other, that the paper size leads to a good “form factor” for your book, and is appropriate given the kind of binding used, etc.

• and finally, that after you have carefully proofread and verified every line of every page, the reader will not see something totally different, with lines of different widths broken in different places, etc.

Or if you cannot relate to that example, then forget all that, because it's just a special case of a simpler, more general case: suppose you know that your document is ultimately going to be read on paper, and you'd like to make sure it can look the same ten years from now as it does today.

Then PDF (especially PDF/A) is a decent format for this case.

(PS: I've seen very few websites that have good typography in the sense that when printed they approach anything like the quality of a halfway decent book.)

Re: A Python Library to extract tabular data from PDFs

#87

Post by worldexplorer »

radarsat1 wrote: post #40

> When using Stream, tables aren’t autodetected. Stream treats the whole page as a single table I've often wondered if image semantic segmentation methods as used in the ML community could successfully identify things like "there is a table (or figure) here, it's not part of the main text". I mean, it seems that humans should be able to do this even without reading the text so I don't see why a CNN couldn't.

Yes it should work. Definitely worth trying.

Re: A Python Library to extract tabular data from PDFs

#88

Post by xfer »

throwaway287391 wrote: post #85
aasasd wrote: post #80

Earlier quoted context omitted.

For some reason I have to battle the same argument again and again in this thread. It's like people are blinded from having seen glamour magazines with crazy layouts. The solution is that you don't use fancy layouts for papers. Imagine that you only have HTML 2.0. Put your text in paragraphs, put images and large formulas in separate paragraphs between those of text. Now chuck that into a ready-made styling template…

Okay, I could be convinced that a Markdown-type format with HTML output and very restrictive formatting could work, as long as it really includes (nearly) all of the support for tables, images, and math notation that LaTeX does, and displays well in any reasonable browser/platform. But is there an implemented working compiler of such a format? I sort of doubt it. (But if I'm wrong, please do post it.) Going back to R…

You should look at several papers written in scribble(racket doc templating language), they are beautiful, not hard to author(compared to latex).

Here is an example: http://www.cs.utah.edu/plt/scope-sets .

Although some of the models on semantics are pictures generated from latex :p

Re: A Python Library to extract tabular data from PDFs

#89

Post by aasasd »

bosie wrote: post #84
aasasd wrote: post #79

Earlier quoted context omitted.

“Resize” is an ambiguous term, so you may or may not have understood me correctly, I'm still not sure. My (primary) problem with PDF is that it doesn't adapt to displays of different sizes—mobile, electroink and tablet devices in addition to desktop machines—and can't be reformatted on a device (e.g. to adjust the author's typesetting choices). Did you mean “zooming” the page in/out on the same device? That's not a b…

Fair point about font-resizing. Since you are curious due to professional curiosity: what i meant by resize is the utilization of the device's screen. If my screen allows for a 1200px wide browser window, the main content shouldn't use 800px of it. On my 5000px wide screen, nytimes.com articles seem to utilize a whooping 10-15% (i am guessing). Might as well just send me a fixed-layout PDF. That being said, I doubt i…

Oh, I happen to know a bit about this issue. It's very much not recommended to have long lines of text, as you may already know—because that way the eye has trouble finding the next line when returning from the end of the previous one, and the entire reading endeavor becomes rather janky experience. That's one of the primary reasons that we have book pages in portrait orientation and that newspaper articles are stretched in vertical columns. With this limitation, it would be quite pointless to try “utilizing” the screen area with other elements, since they can't just be arbitrarily hanging around the text.

If you're doing a lot of reading, you would do better by having your screen in portrait orientation. Wide screens are better suited for other tasks.

I'm tempted to note, however, that HTML with a simple layout, again, can technically be hammered into displaying in several columns on a wide screen. You'd probably want/need site-specific solutions if you want to keep the site's navigation. But if you need only the main content, you could use an extension akin to the “reading mode” of Firefox/Safari/Pocket, and override the CSS to break content into columns. (There might also be such extensions around that already have columns built in.)

Re: A Python Library to extract tabular data from PDFs

#90

Post by danso »

aasasd wrote: post #61
danso wrote: post #54

Earlier quoted context omitted.

PDF is for printing and for (well, with the exception of a few edge cases) guaranteeing a layout and display given a set of paper dimensions. HTML has the advantage of responsiveness, but the inherent problem of variable output. When I was a professor and advising my students on creating portfolios, I told them to build websites of course. But I told them to also have a link to a one-page PDF because many organizatio…

Variable output is not a problem, it's exactly what's needed. The days of standard paper formats are over, old man: in a decade people will have documents delivered straight to their retinas, or read into their ears—but everyone will still have to scroll PDFs back and forth with no possibility of reformatting, because lots of papers are published in it. If you want to have your document printed nicely, just prepare i…

No, variable output is not "exactly what's needed". Layout of information is an actual skill -- whether it's a resume, a newspaper front page, or a photo gallery -- and we can expect layout to be an important design factor as long as humans have eyes that can process information in formats other than a byte stream.

HTML has been an excellent format for delivering data and information across innumerable devices and visual dimensions. That adaptability comes with tradeoffs. As others have pointed out, anyone who's browsed the Internet Archive knows how HTML, beautiful and organized in its own time, can look like slop today. Paper/PDF's tradeoff, of course, is its rigidity.

Post reply on HN