Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

171–180 of 236 posts

Re: So you want to parse a PDF?

#171

Post by crispyambulance »

gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

  > The answer seems obvious to me: [1, 2, 3]
Yeah, that would be nice, but it is SO RARE, I've not even heard of that being possible, let alone how to get at the metadata with godforsaken readers like Acrobat. I mean, I've used pdf's since literally the beginning. Never knew that was a feature.

I think this is all the consequence of the failure of XML and it's promise of its related formatting and transformation tooling. The 90's vision was beautiful: semantic documents with separate presentation and transformation tools/languages, all machine readable, versioned, importable, extensible. But no. Here we are in the year 2025. And what do we got? pdf, html, markdown, json, yaml, and csv.

There are solid reasons why XML failed, but the reasons were human and organizational, and NOT because of the well-thought-out tech.

Re: So you want to parse a PDF?

#172

Post by dotancohen »

throwaway4496 wrote: post #74
koakuma-chan wrote: post #35

Earlier quoted context omitted.

I think it's reasonable because their models are probably trained on images, and not whatever "structured data" you may get out of a PDF.

No model can do better on images than structured data. I am not sure if I am on crack or you're all talking nonsense.

You are assuming structure where there is none. It's not the crack, it's the lack of experience with PDF from diverse sources. Just for instance, I had a period where I was _regularly_ working with PDF files with the letters in reverse order, each letter laid out individually (not a single complete word in the file).

Re: So you want to parse a PDF?

#173

Post by sychou »

Amusing, cringey, and also painful that two of our most common formats - PDF and HTML/CSS/JS - are such a challenge to parse and display. Probably a quarter of AI compute power seems to go into understanding just those two.

Re: So you want to parse a PDF?

#174

Post by crabmusket »

gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

If your solution involves convincing producers of PDFs to produce structured data instead, then do the rest of us a favour and convince them to jettison PDF entirely and just produce the structured data.

PDFs are a social problem, not a technical problem.

Re: So you want to parse a PDF?

#175

Post by ramraj07 »

joakleaf wrote: post #167
reactordev wrote: post #154

Earlier quoted context omitted.

We tried the xml structured route, only to end up with pea soup afterwards. Rasterizing and OCR was the only way to get standardized output.

I know OCR is easier to set up, but you lose a lot going that way. We process several million pages from Newspapers and Magazines from all over the world with medium to very high complexity layouts. We built the PDF parser on top of open source PDF libraries, and this gives many advantages: • We can accurately get headlines other text placed on top on images. OCR is generally hopeless with text placed on top of image…

Do you have your parser released as a service? Curious to test it out.

Re: So you want to parse a PDF?

#176

Post by reactordev »

joakleaf wrote: post #167
reactordev wrote: post #154

Earlier quoted context omitted.

We tried the xml structured route, only to end up with pea soup afterwards. Rasterizing and OCR was the only way to get standardized output.

I know OCR is easier to set up, but you lose a lot going that way. We process several million pages from Newspapers and Magazines from all over the world with medium to very high complexity layouts. We built the PDF parser on top of open source PDF libraries, and this gives many advantages: • We can accurately get headlines other text placed on top on images. OCR is generally hopeless with text placed on top of image…

We were 99.99% accurate with our OCR method. It’s not just vanilla ocr but a couple of extractions of metadata (including the xml from the forms) and textract-like json of the document to perform ocr on the right parts.

A lot has changed in 10 years. This was for a major financial institution and it worked great.

Re: So you want to parse a PDF?

#177

Post by ramraj07 »

throwaway4496 wrote: post #119
kwon-young wrote: post #111

Earlier quoted context omitted.

Just to illustrate this point, poppler [1] (which is the most popular pdf renderer in open source) has a little tool called pdf2cairo [2] which can render a pdf into a svg. This means you can delegate all pdf rendering to poppler and only work with actual graphical objects to extract semantics. I think the reason this method is not popular is that there are still many ways to encode a semantic object graphically. A s…

There is also PDF to HTML, PDF to Text, MuPDF also has PDF to XML, both projects along with a bucketful of other PDF toolkits have PDF to PS, and there is many many XML, HTML, and Text outputs for PS. Rastering and OCR'ing PDF is like using regex to parse XHTML. My eyes are starting to bleed out, I am done here.

It looks like you make a lot of valid points, but also have an extremely visceral reaction because theres a company out there thats using AI in a way that offends you. I mean fair still.

But im a guy who's in the market for a pdf parser service, im happy to pay pretty penny per page processed. I just want a service that works without me thinking for a second about any of the problems you guys are all discussing. What service do I use? Do I care if it uses AI in the lamest way possible? The only thing that matters are the results. There are two people including you in this thread ramming with pdf parsing gyan but from reading it all, it doesn't look like I can do things the right way without spending months fully immersed in this problem alone. If you or anyone has a non blunt AI service that I can use Ill be glad to check it out.

Re: So you want to parse a PDF?

#178

Post by ramraj07 »

MartinMond wrote: post #148
hobs wrote: post #139

Earlier quoted context omitted.

Looks super interesting, except for there's no pricing on the page that I could find except for contact sales - totally understand wanting to do a higher touch sales process, but that's going to bounce some % of eng types who want to try things out but have been bamboozled before.

I know - we're working on adding Self-Serve sign up for non-enterprise deals. But regarding our pricing - I can point you at an actual testimonial https://www.g2.com/products/pspdfkit-sdk/reviews/pspdfkit-sd... > These pricing structures can be complex and NEED to be understood fully before moving forward with purchase. However, out of all of the solutions that I reviewed, [Nutrient] was the one that walked me throug…

I think youre losing a customer because you don't have that option. Im not gonna contact sales and sit through another inane sales pitch zoom call that should be no more than 5 minutes stretched to an hour before I even know if your solution works. And im most definitely not gonna keep fingers crossed the pricing makes sense.

Re: So you want to parse a PDF?

#179

Post by AtNightWeCode »

neuroelectron wrote: post #147
AtNightWeCode wrote: post #138

PDF is a format for preserving layouts across different platforms when viewing and printing. It is not intended for data processing and so on. I don't see why a structured document format can't exist that simplifies processing and increases accessibility while still preserving the layouts.

What about open office docs? (ODF – OpenDocument Format, like .odt, .ods, .odp) JavaScript in particular is actively hostile to stability and determinism.

I have not looked at those formats but take docx for example. That structure is complicated because the layout needs to be described and editable.

Re: So you want to parse a PDF?

#180

Post by fennecfoxy »

pjc50 wrote: post #131
gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

"Should" is doing a lot of heavy lifting here. I think people underestimate how much use of PDF is actually adversarial; starting with using it for CVs to discourage it being edited by middlemen, then "redaction" by drawing boxes over part of the image, encoding tables in PDF rather than providing CSV to discourage analysis, and so on.

Yep, HSBC (UK) only does statements in PDF now and not CSV. I'm not sure that they've done this on purpose but it certainly feels like it. I'd like to be able to analyse my statements and even started writing a parser for them but the way they've done it is just so fucked, I gave up out of pure rage and frustration.
Post reply on HN