Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

231–236 of 236 posts

Re: So you want to parse a PDF?

#231

Post by dotancohen »

throwaway4496 wrote: post #227
dotancohen wrote: post #222

Earlier quoted context omitted.

Then I would very much appreciate if you would enlighten me. I'm serious, I would love nothing more than for you to prove your point, teach me something, and win an internet argument. Once rendered, do any of the rendering engines have e.g. a selectable or accessible text? Poppler didn't, neither did some Java library that I tried. For me, learning something new is very much worth losing the internet argument!

I have explained the details in other comments, have a look. But you can start by looking at pdftotext from Poppler, it is ready to go for 60-70% of cases with -layout flag, with bbox-layout you get even more details.

Thank you. Even with box layout one can not even know that there is a coherent word or phrase to extract, without visually inspecting the PDF beforehand. I've been there, fighting with it right in the CLI and finding that there is no way to even progress to a script.

The advantage of the OCR method is that it effectively performs that visual inspection. That's why it is preferable for PDFs of disparate origin.

Re: So you want to parse a PDF?

#232

Post by throwaway4496 »

dotancohen wrote: post #231
throwaway4496 wrote: post #227

Earlier quoted context omitted.

I have explained the details in other comments, have a look. But you can start by looking at pdftotext from Poppler, it is ready to go for 60-70% of cases with -layout flag, with bbox-layout you get even more details.

Thank you. Even with box layout one can not even know that there is a coherent word or phrase to extract, without visually inspecting the PDF beforehand. I've been there, fighting with it right in the CLI and finding that there is no way to even progress to a script. The advantage of the OCR method is that it effectively performs that visual inspection. That's why it is preferable for PDFs of disparate origin.

What kind of semantics can you infer from the text of OCRing a bitmap that you can't infer from the text generated directly from the PDF? Is it the lack of OCR mistakes? The hallucinations? Or something else?

Re: So you want to parse a PDF?

#233

Post by immibis »

vander_elst wrote: post #81
diptanu wrote: post #15

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers. This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs. We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recogn…

It might sound absurd, but on paper this should be the best way to approach the problem. My understanding is that PDFs are intended to produce an output that is consumed by humans and not by computers, the format seems to be focused on how to display some data so that a human can (hopefully) easily read them. Here it seems that we are using a technique that mimics the human approach, which would seem to make sense. I…

It's Portable Document Format, and the Document refers to paper documents, not computer files.

In other words, this is a way to get a paper document into a computer.

That's why half of them are just images: they were scanned by scanners. Sometimes the images have OCR metadata so you can select text and when you copy and paste it it's wrong.

Re: So you want to parse a PDF?

#234

Post by dotancohen »

throwaway4496 wrote: post #232
dotancohen wrote: post #231

Earlier quoted context omitted.

Thank you. Even with box layout one can not even know that there is a coherent word or phrase to extract, without visually inspecting the PDF beforehand. I've been there, fighting with it right in the CLI and finding that there is no way to even progress to a script. The advantage of the OCR method is that it effectively performs that visual inspection. That's why it is preferable for PDFs of disparate origin.

What kind of semantics can you infer from the text of OCRing a bitmap that you can't infer from the text generated directly from the PDF? Is it the lack of OCR mistakes? The hallucinations? Or something else?

In the cases that I've seen, the PDF software does not generate text strings. It generates individual letters. It is up to any application to try to figure out how those individual letters relate to one another.

Re: So you want to parse a PDF?

#235

Post by throwaway4496 »

dotancohen wrote: post #234
throwaway4496 wrote: post #232

Earlier quoted context omitted.

What kind of semantics can you infer from the text of OCRing a bitmap that you can't infer from the text generated directly from the PDF? Is it the lack of OCR mistakes? The hallucinations? Or something else?

In the cases that I've seen, the PDF software does not generate text strings. It generates individual letters. It is up to any application to try to figure out how those individual letters relate to one another.

Did you even read my comment? The "application" is called pdftotext, and instead of putting the individual letters on a bitmap, it puts them in a string literal.

Re: So you want to parse a PDF?

#236

Post by dotancohen »

throwaway4496 wrote: post #235
dotancohen wrote: post #234

Earlier quoted context omitted.

In the cases that I've seen, the PDF software does not generate text strings. It generates individual letters. It is up to any application to try to figure out how those individual letters relate to one another.

Did you even read my comment? The "application" is called pdftotext, and instead of putting the individual letters on a bitmap, it puts them in a string literal.

I do not understand why you insist on being polemic to win an internet argument, when I'm giving you all the tools to win the internet argument by virtue of being correct.

I did read your comment, because my intention here is to learn. I already described how tools such as pdftotext do not produce strings when each letter is positioned independently. I even gave an example of a few replies up.

Post reply on HN