Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

211–220 of 236 posts

Re: So you want to parse a PDF?

#211

Post by gcanyon »

layer8 wrote: post #181
gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

That “obvious solution” is very reminiscent of https://xkcd.com/927/ . And, as a sibling notes, it opens up the failure case of the attached data not matching the rendered PDF contents.

Yeah, I'm not proposing anything new -- just that apps use what's already available: embedding the content of a PDF as JSON, similar, or even plain text.

Re: So you want to parse a PDF?

#212

Post by throwaway4496 »

dotancohen wrote: post #172
throwaway4496 wrote: post #74

Earlier quoted context omitted.

No model can do better on images than structured data. I am not sure if I am on crack or you're all talking nonsense.

You are assuming structure where there is none. It's not the crack, it's the lack of experience with PDF from diverse sources. Just for instance, I had a period where I was _regularly_ working with PDF files with the letters in reverse order, each letter laid out individually (not a single complete word in the file).

You're thinking "rendering structured data" means parsing PDF as text. That is just wrong. Carefully read what I said. You render the PDF, but into structured data rather than raster. If you still get letters in reverse when you render your PDF into structured data, your rendering engine is broken.

Re: So you want to parse a PDF?

#213

Post by throwaway4496 »

bsder wrote: post #40
throwaway4496 wrote: post #20

Earlier quoted context omitted.

How is it reasonable to render the PDF, rasterize it, OCR it, use AI, instead of just using the "quality implementation" to actually get structured data out? Sounds like "I don't know programming, so I will just use AI".

> How is it reasonable to render the PDF, rasterize it, OCR it, use AI, instead of just using the "quality implementation" to actually get structured data out? Because the underlying "structured data" is never checked while the visual output is checked by dozens of people. "Truth" is the stuff that the meatbags call "truth" as seen by their squishy ocular balls--what the computer sees doesn't matter.

Your mistake is in thinking that computers "see the image", second, you somehow think the output of OCR is different from a PDF engine that renders it into structured data/text.

Re: So you want to parse a PDF?

#214

Post by throwaway4496 »

yxhuvud wrote: post #150
throwaway4496 wrote: post #120

Earlier quoted context omitted.

[flagged]

Well, perhaps you are exposed only to special snowflakes of pdfs that are from a single source and somewhat well formed and easy to extract from. Other, like me, are working at companies that also have lots of PDFs, from many, many different sources, and there are no easy ways to extract structured data or even text in a way that always work.

If you actually read what I have been saying and commenting, you would realise how silly your comment is.

Re: So you want to parse a PDF?

#215

Post by throwaway4496 »

troupo wrote: post #207
throwaway4496 wrote: post #99

Earlier quoted context omitted.

Computers "don't understand" things. They process things, and what you're saying is called layoutinng which is a key part of PDF rendering. I do understand for someone unfamiliar with the internals of file formats, parsing, text shapping, and rendering in general, it all might seem like a blackmagic.

No one said it was as black magic. In the context of OCR and parsing PDFs to convert them to structured data and/or text, rendering is a completely different task from text extraction. As people have pointed out many times in the discussion: https://news.ycombinator.com/item?id=44783004 , https://news.ycombinator.com/item?id=44782930 , https://news.ycombinator.com/item?id=44789733 etc.

You're wrong. There is nothing inherent in "rendering" that means "raster or pixels". You can render PDFs or any format into any format you want, including XML for example.

In fact, in majority of PDFs, a large part of rendering has to do with composing text.

Re: So you want to parse a PDF?

#217

Post by dotancohen »

throwaway4496 wrote: post #212
dotancohen wrote: post #172

Earlier quoted context omitted.

You are assuming structure where there is none. It's not the crack, it's the lack of experience with PDF from diverse sources. Just for instance, I had a period where I was _regularly_ working with PDF files with the letters in reverse order, each letter laid out individually (not a single complete word in the file).

You're thinking "rendering structured data" means parsing PDF as text. That is just wrong. Carefully read what I said. You render the PDF, but into structured data rather than raster. If you still get letters in reverse when you render your PDF into structured data, your rendering engine is broken.

How do you render into structured data, from disparate letters that are not structured?

  D10
  E1
  H0
  L2,3,9
  O4,7
  R8
  W6
I'm sure that you could look at that and figure out how to structure it. But I highly doubt that you have a general-purpose computer program that can parse that into structured data, having never encountered such a format before. Yet, that is how many real-world PDF files are composed.

Re: So you want to parse a PDF?

#218

Post by throwaway4496 »

sidebute wrote: post #197
throwaway4496 wrote: post #79

Earlier quoted context omitted.

I would hire someone who understands PDFs instead of doing the equivalent of printing a digital document and scanning it for "digital record keeping". Stop everything and hire someone who understands the basics of data processing and some PDF.

What's the economic justification? Let's assume we have a staff of 10 and they're fully allocated to committed features and deadlines, so they can't be shifted elsewhere. You're the CTO and you ask the BOD for another $150k/y (fully burdened) + equity to hire a new developer with PDF skills. The COB asks you directly: "You can get a battle-tested PDF parser off-the-shelf for little or no cost. We're not in the PDF pa…

The mistake all of you're making is the assumption that PDF rendering means rasteration. Everything else crumbles down from that misconception.

Re: So you want to parse a PDF?

#219

Post by manishsn »

cylemons wrote: post #204
Macha wrote: post #160

Earlier quoted context omitted.

Sometimes the characters aren’t even characters, just paths

Wouldn't that be very space inefficient to repeat the paths every time a letter appears in the file? Or you mean that glyph Ids don't necessarily map to Unicode?

sometimes in fancy articles you might see the first letter is large and ornate which is most likely a path also like you said glyph IDs always don't necessarily map to unicode or the creator can intentionally mangle the 'to unicode' map of Identity-H embedded font in the pdf if he is nasty

Re: So you want to parse a PDF?

#220

Post by throwaway4496 »

dotancohen wrote: post #217
throwaway4496 wrote: post #212

Earlier quoted context omitted.

You're thinking "rendering structured data" means parsing PDF as text. That is just wrong. Carefully read what I said. You render the PDF, but into structured data rather than raster. If you still get letters in reverse when you render your PDF into structured data, your rendering engine is broken.

How do you render into structured data, from disparate letters that are not structured? D10 E1 H0 L2,3,9 O4,7 R8 W6 I'm sure that you could look at that and figure out how to structure it. But I highly doubt that you have a general-purpose computer program that can parse that into structured data, having never encountered such a format before. Yet, that is how many real-world PDF files are composed.

It is called rendering. MuPDF, Poppler, PDFjs, and so on. The problem is that you and everyone else thinks "rendering" means bitmaps. That is not how it works.
Post reply on HN