Parsing PDFs is filed under 'might make me quit on the spot,' depending on the severity of the ask.
So you want to parse a PDF?
201–210 of 236 posts
Re: So you want to parse a PDF?
#202The subsequent article "So you want to PRINT a PDF" is stuck in a queue somewhere.
Well, I say 'stuck' - it actually got timed out of the queue, but that doesn't raise an error so no one knows about it.
Re: So you want to parse a PDF?
#203Earlier quoted context omitted.
Seems like a fairly reasonable decision given all the high quality implementations out there.
How is it reasonable to render the PDF, rasterize it, OCR it, use AI, instead of just using the "quality implementation" to actually get structured data out? Sounds like "I don't know programming, so I will just use AI".
PDF is a list of drawing commands (not exactly but a useful simplification). All those draw commands from some JS lib or in SVG? Or in every other platform's API? PDF or Postscript probably did them first. The model of "there is some canvas in which I define coordinate spaces then issue commands to draw $thing at position $(x,y), scaled by $z".
You might think of your post as a . Some kind of paragraph or box of text in which the text is laid out and styles applied. That's how HTML does it.
PDF doesn't necessarily work that way. Different lines, words, or letters can be in entirely different places in the document. Anything that resembles a separator, table, etc can also be anywhere in the document and might be output as a bunch of separate lines disconnected from both each other and the text. A renderer might output two-column text as it runs horizontally across the page so when you "parse" it by machine the text from both columns gets interleaved. Or it might output the columns separately.
You can see a user-visible side-effect of this when PDF text selection is done the straightforward way: sometimes you have no problem selecting text. In other documents selection seems to jump around or select abject nonsense unrelated to cursor position. That's because the underlying objects are not laid out in a display "flow" the way HTML does by default so selection is selecting the next object in the document rather than the next object by visual position.
Re: So you want to parse a PDF?
#204Earlier quoted context omitted.
PDF is more like a glorified svg format than a word format. It only contains info on how the document should look but so semantic information like sentences, paragraphs, etc. Just a bag of characters positioned in certain places.
Sometimes the characters aren’t even characters, just paths
Wouldn't that be very space inefficient to repeat the paths every time a letter appears in the file? Or you mean that glyph Ids don't necessarily map to Unicode?
Re: So you want to parse a PDF?
#205If microsoft was able to push their docx garbage into being a standard, nothing surprise me any more.
Re: So you want to parse a PDF?
#206Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…
I have started treading everything as images when multimodal LLMs appeared. Even emails. It's so much more robust.
Especially emails are often used as a container to send a PDF (e.g. a contract) that contains an image of a contract that was printed. Very very common.
I have just moved my company's RAG indexing to images and multimodal embedding. Works pretty well.
Re: So you want to parse a PDF?
#207Earlier quoted context omitted.
Rendering is a different problem from understanding what's rendered. If your PDF renders a part of the sentence at the beginning of the document, a part in the middle, and a part at the end, split between multiple sections, it's still rather trivial to render. To parse and understand that this is the same sentence? A completely different matter.
Computers "don't understand" things. They process things, and what you're saying is called layoutinng which is a key part of PDF rendering. I do understand for someone unfamiliar with the internals of file formats, parsing, text shapping, and rendering in general, it all might seem like a blackmagic.
No one said it was as black magic. In the context of OCR and parsing PDFs to convert them to structured data and/or text, rendering is a completely different task from text extraction.
As people have pointed out many times in the discussion: https://news.ycombinator.com/item?id=44783004, https://news.ycombinator.com/item?id=44782930, https://news.ycombinator.com/item?id=44789733 etc.
Re: So you want to parse a PDF?
#208Earlier quoted context omitted.
While you're doing this, please also tell people to stop producing PDF files in the first place, so that eventually the number of new PDFs can drop to 0. There's no hope for the format ever since manager types decided that it is "a way to put paper in the computer" and not the publishing intermediate format it was actually supposed to be. A vague facsimile of digitization that should have never taken off the way it d…
PDFs serve their purpose well. Except for some niche open source Linux tools, they render the same way in every application you open them in, in practically every version of that application. Unlike document formats like docx/odf/tex/whatever files that reformat themselves depending on the mood of the computer on the day you open them. And unlike raw image files, you can actually comfortably zoom in and read the text…
You don't need the exact flowing of text to be consistent, outside of publishing. This is an anti-feature most of the time, something you specifically don't want.
Zooming is not something PDFs do well at all. I'm not sure in what universe you could call this a usability benefit. Just because it's made of vector graphics doesn't mean you've implemented zoom in a way that is actually usable. People with poor vision (who cannot otherwise use eyeglasses) don't use a magnifying glass, they use the large-print variant of a document. Telling them to use a magnifying glass would be saying "no, we did not accommodate for low eyesight at all, deal with it".
Re: So you want to parse a PDF?
#209The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…
So what you're saying is: the solution to PDF parsing is make a new file format altogether lol. Very helpful.
Not at all. PDFs support embedded content, and JSON (or similar) is a fine way to store that content. So is plain text if it comes to it.
Re: So you want to parse a PDF?
#210The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…
How would that work for a scan of a handwritten document or similar, assuming scanners / consumer computers don't have perfect OCR?
It wouldn't, of course.