Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I am building a new Python library to read/write PDF files

github.com

Post reply on HN
Print view View original

21–30 of 126 posts

Re: Show HN: I am building a new Python library to read/write PDF files

#21

Post by jahewson »

newsclues wrote: post #15
jeremynixon wrote: post #11

Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?

PDF has always seemed to be a janky Adobe product. Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?

That would be XPS https://en.m.wikipedia.org/wiki/Open_XML_Paper_Specification

Re: Show HN: I am building a new Python library to read/write PDF files

#22

Post by macintux »

mdaniel wrote: post #12
password4321 wrote: post #2

Is there a list of open source PDF libraries for various languages? And related: the best tools to generate PDFs from HTML.

"list" is probably harsh, but I've had good luck trawling through the GitHub topics to find such things: https://github.com/search?p=1&q=topic%3Apdf&type=Repositorie... Makes me miss freshmeat.net which would have been my answer a few years ago (freshcode.club just isn't the same, although bless them for trying)

Circa 25 years ago I was newly wed and my wife happened to be watching over my shoulder as I looked up something online. I started typing "fr" and "freshmeat.net" came up as a possible completion.

She was, to put it mildly, immediately suspicious of my browsing habits.

Re: Show HN: I am building a new Python library to read/write PDF files

#23

Post by macintux »

newsclues wrote: post #15
jeremynixon wrote: post #11

Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?

PDF has always seemed to be a janky Adobe product. Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?

It does seem like that would be a good opportunity to weed out some of the insecure aspects of the format.

Unfortunately in practice it would mean that everyone would have to support both PDF and PDF2.

Re: Show HN: I am building a new Python library to read/write PDF files

#24

Post by steampilot »

newsclues wrote: post #15
jeremynixon wrote: post #11

Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?

PDF has always seemed to be a janky Adobe product. Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?

I think it's not too late to create a modern open-source alternative to PDF. I find it unacceptable that something that has become so widely used doesn't have proper free tools for editing. Society shouldn't be limited by income if they want (have?) to use PDFs, or else suffer from a bad experience. The other bigger problem with PDF is that a lot of the times it's used for something for which it wasn't made to be used for. Anything that is expected to be consumed on both mobile and desktop devices should never use PDFs. Government forms should not use PDFs with hacky embedded scripts either.

Re: Show HN: I am building a new Python library to read/write PDF files

#25

Post by dbrueck »

jeremynixon wrote: post #11

Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?

Others have given some good answers already, but I'll add one more: PDF is all about creating a final layout, but given a final layout, there are an infinite number of inputs that could have produced it, but if you are parsing the PDF, most of the time you are trying to get back something higher level than e.g. a command to draw a single character at some X,Y location. But many PDFs in the wild were not generated with that type of semantic extraction in mind, so you have to sort of fudge things to get what you want, and that's where it becomes complex and crazy.

For example, I once had to try to parse PDF invoices generated by some legacy system, and at the bottom was a line that read something like, "Total: $32.56". But in the PDF there was an instruction to write out the string "Total:" and a separate one to write out the amount string, but there was nothing in the PDF itself that correlated the two in any way at all (they didn't appear anywhere close to either other in the page's hierarchy, they weren't at a fixed set of coordinates, etc, etc.).

Re: Show HN: I am building a new Python library to read/write PDF files

#26

Post by programmarchy »

I’ve done a bit of PDF wrangling in Python, so figured I’d describe the lay of the land.

PyPDF [1] is great for reading and writing PDF files, especially dealing with pages, but it’s not great for generating paths, shapes, graphics, etc.

However, reportlab [2] has a great API for generating those things, but is lacking in the file IO and page management department. But the content streams it generates can be plugged into PyPDF pretty easily.

Finally, there’s pdfplumber which does an amazing job of parsing tabular data from PDF structures, and pytesseract which can perform OCR on PDFs that are actually just image data rather than structured data.

There’s not really a one-stop-shop for PDFs, but some pretty good tools that can be combined to get the job done.

Will be curious to see how this project develops!

[1] https://pypi.org/project/PyPDF2/

[2] https://pypi.org/project/reportlab/

Re: Show HN: I am building a new Python library to read/write PDF files

#27

Post by mythrwy »

999900000999 wrote: post #18
jeremynixon wrote: post #13

Earlier quoted context omitted.

What libraries do you see as being SOTA? Fitz? Tika? My hope is that computer vision + OCR will solve this once and for all in near future.

OCR is built into Adobe's PDF reader, issue is it's 15$ a month. I really want to see OCR become easier to use, but I don't know why it's such a hard problem in the first place.

There is the python library ocrmypdf https://ocrmypdf.readthedocs.io/en/latest/ that works really well. I have found the results comparable to Adobe in accuracy.

I believe it uses tesseract, ghostscript and some other libraries.

Speaking of ghostscript, one way to deal with problematic PDFs is to print them to file and deal with the result instead.

Re: Show HN: I am building a new Python library to read/write PDF files

#28

Post by mdaniel »

newsclues wrote: post #15
jeremynixon wrote: post #11

Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?

PDF has always seemed to be a janky Adobe product. Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?

I know this is likely a case of "you know what I meant," but there already is a PDF 2.0: https://www.pdfa.org/resource/iso-32000-pdf/

Re: Show HN: I am building a new Python library to read/write PDF files

#29

Post by mythrwy »

chazeon wrote: post #17

On the field of PDF parsing, I think the most interesting project I encountered is pdfquery[1], where the PDF is parsed as a XML tree and you can use XPath to query it. You might encounter rough edges when put it into production work but the idea is like “how come i never thought of this” because PDF has some tree like structure and should be a straightforward solution. [1]: https://github.com/jcushman/pdfquery

There is a command line utility (pdf2text) that will also parse the pdf to an XML tree and you can query with XPaths. I found it works well.

https://pdfminersix.readthedocs.io/en/latest/reference/comma...

Post reply on HN