PDF to Text, a challenging problem • Hacker News Board

Re: PDF to Text, a challenging problem

#151

Post by jimjimjim » Wed, May 14, 2025, 4:38 AM UTC

j45 wrote: post #6

bartread wrote: post #3

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a " (anti-)pattern, and especially on sites that use more semantically useful elements, like . Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are s…

PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that…

uh. There is very little XML and the spec is a thousand pages long.

Re: PDF to Text, a challenging problem

#152

Post by msephton » Wed, May 14, 2025, 4:38 AM UTC

fracus wrote: post #125

Why hasn't the PDF standard been replaced or revised to require the text in meta form? Seems like a no brainer.

It's already supported, but it's an optional feature, so it's up to the app/developer/author.

Re: PDF to Text, a challenging problem

#153

Post by wrs » Wed, May 14, 2025, 4:41 AM UTC

marginalia_nu wrote: post #16

wrs wrote: post #11

Since these are statistical classification problems, it seems like it would be worth trying some old-school machine learning (not an LLM, just an NN) to see how it compares with these manual heuristics.

I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.

But if you believe in your manual heuristics enough to ship them, you must already have a body of tests that you're happy with, right?

Also seems like this is a case where generating synthetic data would be a big help. You don't have to use only real-world documents for training, just examples of the sorts of things real-world documents have in them. Make a vast corpus of semi-random documents in semi-random fonts and settings, printed from Word, Pandoc, LaTeX, etc.

Re: PDF to Text, a challenging problem

#154

Post by aorth » Wed, May 14, 2025, 5:05 AM UTC

dstryr wrote: post #119

varunneal wrote: post #71

Earlier quoted context omitted.

I've been hacking away at trying to process PDFs into Markdown, having encountered similar obstacles to OP regarding header detection (and many other issues). OCR is fantastic these days but maintaining a global structure to the document is much trickier. Consistent HTML seems still out of reach for large documents. I'm having half-decent results with Markdown using multiple passes of an LLM to extract document struc…

Give this project a try. I've been using it with promising results. https://github.com/matthsena/AlcheMark

I tried with one PDF and was surprised to see it connect to some cloud service:

  2025-05-14 07:58:49,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443
  2025-05-14 07:58:50,446 - urllib3.connectionpool - DEBUG - https://openaipublic.blob.core.windows.net:443 "GET /encodings/o200k_base.tiktoken HTTP/1.1" 200 361 3922

The project's README doesn't mention that anywhere...

Re: PDF to Text, a challenging problem

#155

Post by selcuka » Wed, May 14, 2025, 5:25 AM UTC

90s_dev wrote: post #63

Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…

A while ago someone asked me a C++ question and I said "sorry, never worked seriously with C++". Then I remembered that I wrote the client code for a private instant messenger in Borland C++ approximately 20 years ago, that was used by thousands of people.

So yeah, it happens.

Re: PDF to Text, a challenging problem

#156

Post by dqv » Wed, May 14, 2025, 5:39 AM UTC

theamk wrote: post #137

hallman76 wrote: post #136

Earlier quoted context omitted.

We will never get back the collective man-decades of time that has been burned by this format. When will the madness stop?

When we get an alternative that can: (1) be stored in a single file (2) Allow tables, images and anything else that can be shown on a piece paper (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper (4) won't require Javascript or access to external sites that means never.. We've got lucky we at least got PDF before "web designers" made (3) impossible, and marketers made (4…

> (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper

> (4) won't require Javascript or access to external sites

So about that... https://opensource.adobe.com/dc-acrobat-sdk-docs/library/jsa...

Re: PDF to Text, a challenging problem

#157

Post by gpvos » Wed, May 14, 2025, 5:49 AM UTC

theamk wrote: post #137

hallman76 wrote: post #136

Earlier quoted context omitted.

We will never get back the collective man-decades of time that has been burned by this format. When will the madness stop?

When we get an alternative that can: (1) be stored in a single file (2) Allow tables, images and anything else that can be shown on a piece paper (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper (4) won't require Javascript or access to external sites that means never.. We've got lucky we at least got PDF before "web designers" made (3) impossible, and marketers made (4…

Postscript fits that bill better.

Re: PDF to Text, a challenging problem

#158

Post by degamad » Wed, May 14, 2025, 6:26 AM UTC

aorth wrote: post #154

dstryr wrote: post #119

Earlier quoted context omitted.

Give this project a try. I've been using it with promising results. https://github.com/matthsena/AlcheMark

I tried with one PDF and was surprised to see it connect to some cloud service: 2025-05-14 07:58:49,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443 2025-05-14 07:58:50,446 - urllib3.connectionpool - DEBUG - https://openaipublic.blob.core.windows.net:443 "GET /encodings/o200k_base.tiktoken HTTP/1.1" 200 361 3922 The project's README doesn't mention that…

The project's README mentions that it uses tiktoken[0], which is a separate project created by OpenAI.

tiktoken downloads token models the first time you use them, but it does not mention that. It does cache the models, so you shouldn't see more of those connections, if I'm understanding the code correctly.

[0] https://github.com/openai/tiktoken>

Re: PDF to Text, a challenging problem

#159

Post by gpvos » Wed, May 14, 2025, 6:35 AM UTC

normie3000 wrote: post #134

1vuio0pswjnm7 wrote: post #76

Earlier quoted context omitted.

"PDF" is an acronym for for "Portable Document Format" "2.3.2 Portability A PDF file is a 7-bit ASCII file, which means PDF files use only the printable subset of the ASCII character set to describe documents even those with images and special characters. As a result, PDF files are extremely portable across diverse hardware and operating system environments." https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstanda…

> PDF files use only the printable subset of the ASCII character set to describe documents even those with images and special characters Great, so PDF source code is easily printable?

Except most are compressed or contain binary streams. You can transform any PDF into an equivalent ASCII PDF though, e.g. using qpdf.

Re: PDF to Text, a challenging problem

#160

Post by numpad0 » Wed, May 14, 2025, 7:09 AM UTC

theamk wrote: post #137

hallman76 wrote: post #136

Earlier quoted context omitted.

We will never get back the collective man-decades of time that has been burned by this format. When will the madness stop?

When we get an alternative that can: (1) be stored in a single file (2) Allow tables, images and anything else that can be shown on a piece paper (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper (4) won't require Javascript or access to external sites that means never.. We've got lucky we at least got PDF before "web designers" made (3) impossible, and marketers made (4…

(-1) be vector format that never gets pixelated

(0) that reproduce everywhere on any OS perfectly

(0.5) that supports (everything) any typographical engineers ever wanted past and future

Bitmap formats are out from clause -1, Office file formats disqualify from clause 0, Markdown doesn't satisfy clause 0.5. Otherwise a Word .doc format covers most of clauses 1-4.