An alternative: https://github.com/jsvine/pdfplumber
A Python Library to extract tabular data from PDFs
51–60 of 102 posts
Re: A Python Library to extract tabular data from PDFs
#52Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coo…
If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.
PDF (especially full spec'd PDF) seems bad, but I never like reading epub or other formats. PDFs are the static builds of documents.
Re: A Python Library to extract tabular data from PDFs
#53Great work and write up! HN submissions about PDF extraction seem to be as reliably popular as threads mentioning bees or bashing Mongo, which I guess goes to show how pervasive a problem it is.
Re: A Python Library to extract tabular data from PDFs
#54Earlier quoted context omitted.
HTML and PDF are used for different purposes. PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that. HTML is markup for presenting documents application dependent manner.
They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…
PDF is for printing and for (well, with the exception of a few edge cases) guaranteeing a layout and display given a set of paper dimensions. HTML has the advantage of responsiveness, but the inherent problem of variable output.
When I was a professor and advising my students on creating portfolios, I told them to build websites of course. But I told them to also have a link to a one-page PDF because many organizations (not just academia) forward resumes within an organization to someone senior who eventually prints it out. And you don't want that person's first impression be whatever your website's print.css churns out.
Re: A Python Library to extract tabular data from PDFs
#55So many times I have wanted to get this type of data. Visa would send reporting this way and it would have to be manually copied over. They offered CSV but there were extra charges associated. There were some pretty good libraries for paragraph text extraction but the graphs were too tough to deal with.
Re: A Python Library to extract tabular data from PDFs
#56Can't wait to try this out with Percollate!
Re: A Python Library to extract tabular data from PDFs
#57You should also check out pdf.js:
We use it in Polar:
for our PDF management.
It's a pretty robust library and it renders everything on canvas BUT you also get the raw text in the DOM so you can play with it more as an API for managing PDFs.
REALLY nice to be able to use web standards when working with pdf.js.
The downside is that the graphics are rendered to canvas so you're only really getting an image.
Re: A Python Library to extract tabular data from PDFs
#58Earlier quoted context omitted.
HTML and PDF are used for different purposes. PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that. HTML is markup for presenting documents application dependent manner.
They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…
While I agree that PDFs are at times cumbersome to use, I can't think of a valid solution to replace them.
- Fixed layout seems much easier to handle than dynamic layouts. I.e. I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long). And without handling this properly, most of the arguments against PDF usage seem to go out the window.
- I don't know of any way of highlighting, annotating, drawing on an HTML page reliably over multiple devices. Sure, something can be built on but it requires special software, still.
- How do i send someone an HTML copy of a PDF as a single file? (embedded fonts, images etc)
Re: A Python Library to extract tabular data from PDFs
#59Earlier quoted context omitted.
If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.
I think it's probably Stockholm syndrome but I quite enjoying digging into PDFs. There are actually other ways of embedding the text where you have more information to go on. "Characters at coordinates" is a particularly rough hand to be dealt in a PDF (though it definitely happens).
I have been recently writing a library in go to fill out AcroForms, and really, the PDF spec and syntax isn't really that bad or hard to understand. Most everything is a dictonary object. I've seen waaaay worse over-abused OOP monstrosities of a code base in C# and Java
PDF 2.0 (ISO 32000-2) has been out for a while and supposedly it has eliminated a lot of the cruft from the spec. I just wish it was open like PDF 1.7
Re: A Python Library to extract tabular data from PDFs
#60Earlier quoted context omitted.
They certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which…
While I agree that PDFs are at times cumbersome to use, I can't think of a valid solution to replace them. - Fixed layout seems much easier to handle than dynamic layouts. I.e. I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long). And without handling this properly, most of the a…
> I can't recall any website that resizes the content correctly (correctly meaning i see the image within X% of scrolling of the referenced location; that doesn't just make the lines super-long)
I rarely have anything like that happen, so not even sure if I know the exact problem that you have in mind. As far as I can tell, it's specific to when authors put images somewhere distant to the text that mentions them in the one dimension of text flow, e.g. on the next page, or floating in a separate column from the text. The solution is, don't put images far from the text. HTML obviously requires a different approach from PDF: you don't think in terms of two-dimensional physical layout, you think in the one dimension of semantic layout. Most popular content sites are laid out that way now, and I mostly have no problem reading on the desktop or the phone.
> I don't know of any way of highlighting, annotating, drawing on an HTML page reliably over multiple devices. Sure, something can be built on but it requires special software, still.
It requires software just as PDF requires it. Such software isn't ubiquitous precisely because people don't see HTML annotation as a market. It's a typical chicken-and-egg market problem.
To annotate HTML, you abandon the two-dimensional graphical approach just as you do it when producing the document. Instead, you highlight text in paragraphs and attach annotations and drawings to it, independently of the current rendering of the document. Any word processor allows you to highlight text in lines and paragraphs, you do the same thing here. Evernote's web clipper highlights HTML just fine.
> How do i send someone an HTML copy of a PDF as a single file? (embedded fonts, images etc)
You use a format that packs HTML with images, styling and fonts—e.g. MAFF. Come on, it's not rocket science to store what the server sends to the browser. Again, Evernote stores pages fine and could be used for sharing (if the program didn't go to crap overall). It's the same chicken-and-egg problem.