Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Library to extract tabular data from PDFs

blog.socialcops.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: A Python Library to extract tabular data from PDFs

#71

Post by aasasd »

nabla9 wrote: post #68
aasasd wrote: post #64

Earlier quoted context omitted.

HTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format. > Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. Not enough how? What is there that you need besides what the server hands to you, if that's what rendered in the first place? What magical compression methods do you have…

> HTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format. HTML is not a good format and standard for that purpose. It's loose best effort markup with no good consensus on semantics. HTML with images is not good option for papers which have many equations. EPUB3 is emerging standard for what you want, but it's not really good complete solution that can replace PDF/A or…

> It's loose best effort markup with no good consensus on semantics.

And PDF has good semantics? Are we still on the topic of how HTML is better than PDF, or…? We're in the comments for a page that says that PDF tables are characters just floating in space, and people are saying most PDFs out there don't have semantic markup. Meanwhile HTML had semantics efforts for decades now, just choose your flavor.

Blind people read HTML, you know. Do they read PDFs?

> HTML with images is not good option for papers which have many equations.

There's MathML for that, and IIRC other formats too. You could even have embedded TeX like Anki has. Use SVG for fallback.

>> Have a static HTML version that's rendered the same in the future

> We don't have that.

Ooh, chicken-and-egg again? Freeze any of the versions from the past decade with the rendering standards, and you'll have it.

But actually, it doesn't even matter, just like HTML 2.0 can be rendered fine on modern devices (aside from the different text size). Treat your paper as a paper instead of a webzine, don't use crazy layouts, just do “text, image, text” which you'll want anyway for the different displays—and your document will render fine in the future when it will be delivered straight to the retina, instead of making me scroll the PDF back and forth because no reflow.

Re: A Python Library to extract tabular data from PDFs

#74

Post by taylorwc »

catacombs wrote: post #73

A Python library to do this is cool, but there's already Tabula: https://tabula.technology/

They address Tabula in the post:

>The first tool that we tried was Tabula, which has nice user and command-line interfaces, but it either worked perfectly or failed miserably. When it failed, it was difficult to tweak the settings — such as the image thresholding parameters, which influence table detection and can lead to a better output.

Re: A Python Library to extract tabular data from PDFs

#75

Post by throwaway287391 »

aasasd wrote: post #29
spaceandshit wrote: post #26

Earlier quoted context omitted.

And replaced with what?

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

As somebody who's written papers for scientific conferences and journals, I can tell you that this just isn't going to work. I'm not a designer and making a LaTeX-generated PDF with few frills look decent is enough work already. Generating HTML and having to verify that it looks good on every major browser whether it's a mobile or desktop device is just completely out of the question.

Seriously, think about it -- plenty of "professional" webpages with full time designers and UX engineers look like shit on mobile or Firefox or whatever (as you'll know if you've ever read reddit/HN comments). Imagine what a shitshow HTML papers from academics who are desperately doing whatever they can to get a readable version of their paper formatted during the 20 minutes before the deadline would be.

Re: A Python Library to extract tabular data from PDFs

#76

Post by ezequiel-garzon »

aasasd wrote: post #67
svat wrote: post #47

Earlier quoted context omitted.

The great thing about PDF, even the reason for its existence and adoption, is that a (valid) PDF file will look exactly the same — the same characters in the same fonts at exactly the same positions on every page — on any printer or display across the world, and across time. With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices o…

Yeah, except the devices are different, so there's no point in trying to use the same page size on them, you only torture the reader. > With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers. Only if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal. Use a simple ‘te…

Only if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal.

Actually, the plainest CSS-free HTML renders inexplicably small in some modern flagship phones... I mean the proprietary viewport meta tag, which is in the process of becoming CSS: https://www.w3.org/TR/css-device-adapt-1/

Re: A Python Library to extract tabular data from PDFs

#77

Post by bosie »

aasasd wrote: post #66
bosie wrote: post #63

Earlier quoted context omitted.

> I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind. Not even trying to be funny but do you mind sharing some websites that dynamically resize content correctly? I just checked a couple of the usual suspects (reuters, nytimes, guardian, github) and none do it. They are all using (semi-)fixed layouts.

I have the opposite problem of finding a page that could be problematic. Flipped through several articles on those sites, and they all use the linear article layout. Remember that we're talking about publication of static papers, so you look at the main content column on a page, since that's what should be there in a paper. In the main column, those sites use the simple linear flow: ‘text, image, text’—with images oc…

Sorry, i misunderstood you. I thought you wanted to move away from PDFs because they don't resize. But none of the example i gave resize either (neither does pocket or instapaper).

Re: A Python Library to extract tabular data from PDFs

#78

Post by jacquesm »

vortex_ape wrote: post #25
jacquesm wrote: post #23

Oh that is so timely. I've been putting that part of a pipeline I built off for a while due to the complexity and now I can just plug this in. Super neat. Thank you very much!

What does this pipeline do and what software have you used to implement it? I have used Airflow in the past to create ETL pipelines, and plugged in Camelot in one of them to extract tables from PDFs. I also wrote a blog post about it in case you might be interested. https://hackernoon.com/how-to-create-a-workflow-in-apache-ai...

Compress scientific papers.

Thank you for the pointers!

Re: A Python Library to extract tabular data from PDFs

#79

Post by aasasd »

bosie wrote: post #77
aasasd wrote: post #66

Earlier quoted context omitted.

I have the opposite problem of finding a page that could be problematic. Flipped through several articles on those sites, and they all use the linear article layout. Remember that we're talking about publication of static papers, so you look at the main content column on a page, since that's what should be there in a paper. In the main column, those sites use the simple linear flow: ‘text, image, text’—with images oc…

Sorry, i misunderstood you. I thought you wanted to move away from PDFs because they don't resize. But none of the example i gave resize either (neither does pocket or instapaper).

“Resize” is an ambiguous term, so you may or may not have understood me correctly, I'm still not sure. My (primary) problem with PDF is that it doesn't adapt to displays of different sizes—mobile, electroink and tablet devices in addition to desktop machines—and can't be reformatted on a device (e.g. to adjust the author's typesetting choices).

Did you mean “zooming” the page in/out on the same device? That's not a big issue, in my experience: I zoom in on almost every page due to myopia, and rarely have problems. I adjust text properties on mobile devices too, namely in Pocket and e-book readers (which use HTML under the hood these days). Technically, HTML can be rendered with a rigid layout and just be zoomed in/out like a static image—it's a question of the client having this function, or, I think, can be done via a simple CSS property.

If that's still not what you had in mind, I'd like to know what you mean by “resize,” out of professional curiosity.

Re: A Python Library to extract tabular data from PDFs

#80

Post by aasasd »

throwaway287391 wrote: post #75
aasasd wrote: post #29

Earlier quoted context omitted.

With Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.

As somebody who's written papers for scientific conferences and journals, I can tell you that this just isn't going to work. I'm not a designer and making a LaTeX-generated PDF with few frills look decent is enough work already. Generating HTML and having to verify that it looks good on every major browser whether it's a mobile or desktop device is just completely out of the question. Seriously, think about it -- ple…

For some reason I have to battle the same argument again and again in this thread. It's like people are blinded from having seen glamour magazines with crazy layouts.

The solution is that you don't use fancy layouts for papers. Imagine that you only have HTML 2.0. Put your text in paragraphs, put images and large formulas in separate paragraphs between those of text. Now chuck that into a ready-made styling template that applies modern typographic conventions. Voila, you have a great-looking article that can be read on a display of any size, be that today or three decades from now. It can be reformatted into columns or stringed into a horizontal ribbon, printed on paper of any size, read aloud by a text-to-speech engine, saved in apps like Pocket or Evernote.

Most popular content sites today use this layout for the main content column, and the pages can be read fine on phones or saved in apps. Markdown readmes on Github use this layout, and it's smooth sailing with them. Pages in HTML 2.0 from the 90s display just fine on modern devices, aside from the different text size.

You don't need to be a designer or make sure that your articles look fine on different devices if you stick to this simple layout and use tested styling. Pages with full-time designers have problems on mobile devices because those people try to do fancy layouts. Don't use fancy layouts for papers. I've spent zero time fixing problems with layouts in Markdown or, by the way, in posts and comments on sites like Reddit and HN, because they don't allow me to do fancy layouts—and they stay readable on phones. If authors have to spend time fixing layouts of their papers, it's because they use too complex layouts which indeed would have problems displaying on different devices.

Post reply on HN