Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?
Show HN: I am building a new Python library to read/write PDF files
11–20 of 126 posts
Re: Show HN: I am building a new Python library to read/write PDF files
#12Is there a list of open source PDF libraries for various languages? And related: the best tools to generate PDFs from HTML.
"list" is probably harsh, but I've had good luck trawling through the GitHub topics to find such things: https://github.com/search?p=1&q=topic%3Apdf&type=Repositorie...
Makes me miss freshmeat.net which would have been my answer a few years ago (freshcode.club just isn't the same, although bless them for trying)
Re: Show HN: I am building a new Python library to read/write PDF files
#13I wish you all the best! This space has a lot of stuff in it and they’re lacking in some aspect. And that’s not a admonishment, PDF is such a complicated format that there will never be a library that doesn’t come with asterisks — it’s just a matter of picking the thing you want your library to focus on and be good at and you can pretty easily be someone’s favorite lib.
My understanding is that this is largely because you're fighting an adversarial format provider in Adobe, I've read a few papers and journal entries on file format polyglotting, with some focus on PDF and approaches are constantly shifting in nature due to Adobe mooting pathways to success, I think it's partly for security and also IMO partly for obscurity as PDF is a horrific format in all reality except for human v…
What libraries do you see as being SOTA? Fitz? Tika?
My hope is that computer vision + OCR will solve this once and for all in near future.
Re: Show HN: I am building a new Python library to read/write PDF files
#14Is there a list of open source PDF libraries for various languages? And related: the best tools to generate PDFs from HTML.
The best generator is Prince XML but it can be expensive.
Re: Show HN: I am building a new Python library to read/write PDF files
#15Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?
PDF has always seemed to be a janky Adobe product.
Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?
Re: Show HN: I am building a new Python library to read/write PDF files
#16Earlier quoted context omitted.
My understanding is that this is largely because you're fighting an adversarial format provider in Adobe, I've read a few papers and journal entries on file format polyglotting, with some focus on PDF and approaches are constantly shifting in nature due to Adobe mooting pathways to success, I think it's partly for security and also IMO partly for obscurity as PDF is a horrific format in all reality except for human v…
What libraries do you see as being SOTA? Fitz? Tika? My hope is that computer vision + OCR will solve this once and for all in near future.
To be 100% honest it's been a while since I looked into libraries for it, so I couldn't say.
Your second comment rings true, and in my opinion, we are there. Highly recommend throwing some PDFs at AWS Textract and checking out the quality, it wasn't there a few years ago, can safely state it's there now though. I threw stuff at it that previously would just spit out trash, and it handled it fairly well, specifically for table data extraction (I was looking at public stock market quarterly reports).
Cost is the kicker for me, 1000 pages for $15, adds up fairly quickly at any sort of scale!
Re: Show HN: I am building a new Python library to read/write PDF files
#17On the field of PDF parsing, I think the most interesting project I encountered is pdfquery[1], where the PDF is parsed as a XML tree and you can use XPath to query it. You might encounter rough edges when put it into production work but the idea is like “how come i never thought of this” because PDF has some tree like structure and should be a straightforward solution.
Re: Show HN: I am building a new Python library to read/write PDF files
#18Earlier quoted context omitted.
My understanding is that this is largely because you're fighting an adversarial format provider in Adobe, I've read a few papers and journal entries on file format polyglotting, with some focus on PDF and approaches are constantly shifting in nature due to Adobe mooting pathways to success, I think it's partly for security and also IMO partly for obscurity as PDF is a horrific format in all reality except for human v…
What libraries do you see as being SOTA? Fitz? Tika? My hope is that computer vision + OCR will solve this once and for all in near future.
OCR is built into Adobe's PDF reader, issue is it's 15$ a month.
I really want to see OCR become easier to use, but I don't know why it's such a hard problem in the first place.
Re: Show HN: I am building a new Python library to read/write PDF files
#19Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?
Despite the thousands of pages of ISO 32000, the reality is that the format is not defined. Acrobat tolerates unfathomably malformed PDF files generated by old software that predates the opening-up of the standard when people were reverse-engineering it. There’s always some utterly insane file that Acrobat opens just fine and now you get to play the game of figuring out how Acrobat repaired it.
Plus all the fun of the fact that you can embed the following formats inside a PDF:
PNG, JPEG (including CMYK), JPEG 2000 (dead), JBIG2 (dead), CCIT G4 (dead, fax machines), PostScript Type1 fonts (dead), PostScript Type3 fonts (dead), PostScript CIDFonts (pre-Unicode, dead), CFF fonts (the inside of an OTF), TrueType fonts, ICC Profiles, PostScript functions defining Color spaces, XML forms (the worst), LZ compressed data, Run-length compressed data, Deflate-compressed data.
All of which Acrobat will allow to be malformed in various non-standard ways so you need to write your own parsers.
Note the lack of OpenType fonts, also lack of proper Unicode!
Re: Show HN: I am building a new Python library to read/write PDF files
#20Why is the state of the art in PDF parsing SO BAD? This is an incredibly common and important problem. Tika and fitz have very poor results. What is the reason that this is still so backwards?
My current company uses ML to parse PDF invoices and identify fraud. I have no idea how the devs manage this black magic wizardry because they also spend time contributing to infra code before they hired more people like me on board. If anyone wants a great startup idea, look to solving a problem involving parsing PDFs en masse. Maybe something in legal tech. That market is absolutely ripe for disruption.