Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

21–30 of 236 posts

Re: So you want to parse a PDF?

#21

Post by HocusLocus »

Thanks kindly for this well done and brave introduction. There are few people these days who'd even recognize the bare ASCII 'Postscript' form of a PDF at first sight. First step is to unroll into ASCII of course and remove the first wrapper of Flate/ZIP,LZW,RLE. I recently teased Gemini for accepting .PDF and not .EPUB (chapterized html inna zip basically, with almost-guaranteed paragraph streams of UTF-8) and it lamented apologetically that its pdf support was opaque and library oriented. That was very human of it. Aside from a quick recap of the most likely LZW wrapper format, a deep dive into Lineariziation and reordering the objects by 'first use on page X' and writing them out again preceding each page would be a good pain project.

UglyToad is a good name for someone who likes pain. ;-)

Re: So you want to parse a PDF?

#22

Post by throwaway4496 »

diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

This is the parallel of some of the dotcom peak absurdities. We are in the AI peak now.

Re: So you want to parse a PDF?

#23

Post by throwaway4496 »

Alex3917 wrote: post #19
diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

> This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. One of the biggest benefits of PDFs though is that they can contain invisible data. E.g. the spec allows me to embed cryptographic proof that I've worked at the companies I claim to have worked at within my resume. But a vision-based approach obviously isn't going to be able to capture that.

Cryptographic proof of job experience? Please explain more. Sounds interesting.

Re: So you want to parse a PDF?

#24

Post by simonw »

UglyToad wrote: post #12
simonw wrote: post #6

I convert the PDF into an image per page, then dump those images into either an OCR program (if the PDF is a single column) or a vision-LLM (for double columns or more complex layouts). Some vision LLMs can accept PDF inputs directly too, but you need to check that they're going to convert to images and process those rather than attempting and failing to extract the text some other way. I think OpenAI, Anthropic and…

If you don't have a known set of PDF producers this is really the only way to safely consume PDF content. Type 3 fonts alone make pulling text content out unreliable or impossible, before even getting to PDFs containing images of scans. I expect the current LLMs significantly improve upon the previous ways of doing this, e.g. Tesseract, when given an image input? Is there any test you're aware of for model capabiliti…

I've been trying it informally and noting that it's getting really good now - Claude 4 and Gemini 2.5 seem to do a perfect job now, though I'm still paranoid that some rogue instruction in the scanned text (accidental or deliberate) might result in an inaccurate result.

Re: So you want to parse a PDF?

#25

Post by farkin88 »

UglyToad wrote: post #11
farkin88 wrote: post #8

Great rundown. One thing you didn't mention that I thought was interesting to note is incremental-save chains: the first startxref offset is fine, but the /Prev links that Acrobat appends on successive edits may point a few bytes short of the next xref. Most viewers (PDF.js, MuPDF, even Adobe Reader in "repair" mode) fall back to a brute-force scan for obj tokens and reconstruct a fresh table so they work fine while…

You're right, this was a fairly common failure state seen in the sample set. The previous reference or one in the reference chain would point to offset of 0 or outside the bounds of the file, or just be plain wrong. What prompted this post was trying to rewrite the initial parse logic for my project PdfPig[0]. I had originally ported the Java PDFBox code but felt like it should be 'simple' to rewrite more performantl…

That robustness-vs-throughput trade-off is such a staple of PDF parsing. My guess is that the new path is slower because the recovery scan now always walks the whole byte range and has to inflate any object streams it meets before it can trust the offsets even when the first startxref would have been fine.

The 10k-file test set sounds great for confidence-building. Are the failures clustering around certain producer apps like Word, InDesign, scanners, etc.? Or is it just long-tail randomness?

Reading the PR, I like the recovery-first mindset. If the common real-world case is that offsets lie, treating salvage as the default is arguably the most spec-conformant thing you can do. Slow-and-correct beats fast-and-brittle for PDFs any day.

Re: So you want to parse a PDF?

#26

Post by userbinator »

As someone who has written a PDF parser - it's definitely one of the weirdest formats I've seen, and IMHO much of it is caused by attempting to be a mix of both binary and text; and I suspect at least some of these weird cases of bad "incorrect but close" xref offsets may be caused by buggy code that's dealing with LF/CR conversions.

What the article doesn't mention is a lot of newer PDFs (v1.5+) don't even have a regular textual xref table, but the xref table is itself inside an "xref stream", and I believe v1.6+ can have the option of putting objects inside "object streams" too.

Re: So you want to parse a PDF?

#27

Post by spankibalt »

diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

> "This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world."

Well, to be fair, in many cases there's no way around it anyway since the documents in question are only scanned images. And the hardest problems I've seen there are narrative typography artbooks, department store catalogs with complex text and photo blending, as well as old city maps.

Re: So you want to parse a PDF?

#28

Post by do_not_redeem »

throwaway4496 wrote: post #20
bee_rider wrote: post #17

Earlier quoted context omitted.

Seems like a fairly reasonable decision given all the high quality implementations out there.

How is it reasonable to render the PDF, rasterize it, OCR it, use AI, instead of just using the "quality implementation" to actually get structured data out? Sounds like "I don't know programming, so I will just use AI".

PDFs don't always lay out characters in sequence, sometimes they have absolutely positioned individual characters instead.

PDFs don't always use UTF-8, sometimes they assign random-seeming numbers to individual glyphs (this is common if unused glyphs are stripped from an embedded font, for example)

etc etc

Re: So you want to parse a PDF?

#29

Post by rogerrogerr »

throwaway4496 wrote: post #23
Alex3917 wrote: post #19

Earlier quoted context omitted.

> This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. One of the biggest benefits of PDFs though is that they can contain invisible data. E.g. the spec allows me to embed cryptographic proof that I've worked at the companies I claim to have worked at within my resume. But a vision-based approach obviously isn't going to be able to capture that.

Cryptographic proof of job experience? Please explain more. Sounds interesting.

If someone told me there was cryptographic proof of job experience in their PDF, I would probably just believe them because it’d be a weird thing to lie about.

Re: So you want to parse a PDF?

#30

Post by [deleted] »

throwaway4496 wrote: post #20
bee_rider wrote: post #17

Earlier quoted context omitted.

Seems like a fairly reasonable decision given all the high quality implementations out there.

How is it reasonable to render the PDF, rasterize it, OCR it, use AI, instead of just using the "quality implementation" to actually get structured data out? Sounds like "I don't know programming, so I will just use AI".

[deleted]
Post reply on HN