Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

131–140 of 236 posts

Re: So you want to parse a PDF?

#131

Post by pjc50 »

gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

"Should" is doing a lot of heavy lifting here.

I think people underestimate how much use of PDF is actually adversarial; starting with using it for CVs to discourage it being edited by middlemen, then "redaction" by drawing boxes over part of the image, encoding tables in PDF rather than providing CSV to discourage analysis, and so on.

Re: So you want to parse a PDF?

#132

Post by anthk »

kwon-young wrote: post #111
throwaway4496 wrote: post #93

Earlier quoted context omitted.

You can use an existing readymade renderer to render into structured data instead of raster.

Just to illustrate this point, poppler [1] (which is the most popular pdf renderer in open source) has a little tool called pdf2cairo [2] which can render a pdf into a svg. This means you can delegate all pdf rendering to poppler and only work with actual graphical objects to extract semantics. I think the reason this method is not popular is that there are still many ways to encode a semantic object graphically. A s…

pdftotext from poppler has that without doing juggling with formats.

Re: So you want to parse a PDF?

#133

Post by MartinMond »

diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

Nutrient.io Co-Founder here: We’ve been doing PDF for over 10y. PDF Viewers like Web browsers have to be liberal in what they accept, because PDF has been around for so long, and like with HTML ppl generating files often just iterate until they have something that displays correctly in the one viewer they are testing with.

That’s why we built our AI Document Processing SDK (for PDF files) - basically a REST API service, PDF in, structured data in JSON out. With the experience we have in pre-/post-processing all kinds of PDF files on a structural not just visual basis, we can beat purely vision based approaches on cost/performance: https://www.nutrient.io/sdk/ai-document-processing

If you don’t want to suffer the pain of having to deal with figuring this out yourself and instead focus on your actual use case, that’s where we come in.

Re: So you want to parse a PDF?

#134

Post by jpc0 »

pjc50 wrote: post #131
gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

"Should" is doing a lot of heavy lifting here. I think people underestimate how much use of PDF is actually adversarial; starting with using it for CVs to discourage it being edited by middlemen, then "redaction" by drawing boxes over part of the image, encoding tables in PDF rather than providing CSV to discourage analysis, and so on.

Redaction if only drawing a box over content would not be redaction, I believe that even resulted in some information leakage in the past.

PDFs can be edited, unless they are just embedded images but even then it’s possible.

The selling point of PDFs is “word” documents that get correctly displayed everywhere, ie they are a distribution mechanism. If you want access to the underlying data that should be provided separately as CSV or some other format.

PDFs are for humans not computers. I know the argument you are making is that is not what happens in reality and I sympathise, but the problem isn’t with PDFs but with their users and you can’t fix a management problem with technical.

Re: So you want to parse a PDF?

#135

Post by petesergeant »

joakleaf wrote: post #129
petesergeant wrote: post #89

Earlier quoted context omitted.

> instead of just using the "quality implementation" to actually get structured data out? I suggest spending a few minutes using a PDF editor program with some real-world PDFs, or even just copying and pasting text from a range of different PDFs. These files are made up of cute-tricks and hacks that whatever produced them used to make something that visually works. The high-quality implementations just put the pixels…

We do a lot of parsing of PDFs and basically break the structure into 'letter with font at position (box)' because the "structure" within the PDF is unreliable. We have algorithms that combines the individual letters to words, words to lines, lines to boxes all by looking at it geometrically. Obviously identify the spaces between words. We handle hidden text and problematic glyph-to-unicode tables. The output is simi…

Very interesting. How often do you encounter PDFs that are just scanned pages? I had to make heavy use of pdfsandwich last time I was accessing journal articles.

> quality is higher because we don't depend on vision based text recognition

This surprises me a bit; outside of an actual scan leaving the computer I’d expect PDF->image->text in a computer to be essentially lossless.

Re: So you want to parse a PDF?

#136

Post by Muromec »

viraptor wrote: post #104
sbrother wrote: post #52

Earlier quoted context omitted.

I've built document parsing pipelines for a few clients recently, and yeah this approach yields way superior results using what's currently available. Which is completely absurd, but here we are.

I've done only one pipeline trying parse actual PDF structure and the least surprising part of it is that some documents have top-to-bottom layout and others have bottom-to-top, flipped, with text flipped again to be readable. It only goes worse from there. Absurd is correct.

That means you have to put the text (each infividual letter) into its correct place by rendering pdf, but doesnt justify actual OCR which goes one step further and back by rendering and backguessing the glyphs. But thats just text, tables and structure are also somewhere there to be recovered.

Re: So you want to parse a PDF?

#137

Post by nurettin »

diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

It sounds like a trap coyote would use to catch roadrunner. Does it really have to be so convoluted?

Re: So you want to parse a PDF?

#138

Post by AtNightWeCode »

PDF is a format for preserving layouts across different platforms when viewing and printing. It is not intended for data processing and so on. I don't see why a structured document format can't exist that simplifies processing and increases accessibility while still preserving the layouts.

Re: So you want to parse a PDF?

#139

Post by hobs »

MartinMond wrote: post #133
diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

Nutrient.io Co-Founder here: We’ve been doing PDF for over 10y. PDF Viewers like Web browsers have to be liberal in what they accept, because PDF has been around for so long, and like with HTML ppl generating files often just iterate until they have something that displays correctly in the one viewer they are testing with. That’s why we built our AI Document Processing SDK (for PDF files) - basically a REST API servi…

Looks super interesting, except for there's no pricing on the page that I could find except for contact sales - totally understand wanting to do a higher touch sales process, but that's going to bounce some % of eng types who want to try things out but have been bamboozled before.

Re: So you want to parse a PDF?

#140

Post by mpweiher »

gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

Yes, this works and I do this in a few of my apps.

However, there is the issue of the two representations not actually matching.

Post reply on HN