Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PDF to Text, a challenging problem

marginalia.nu

Post reply on HN
Print view View original

21–30 of 206 posts

Re: PDF to Text, a challenging problem

#21

Post by layer8 »

j45 wrote: post #6
bartread wrote: post #3

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a " (anti-)pattern, and especially on sites that use more semantically useful elements, like . Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are s…

PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that…

> PDFs inherently are a markup / xml format

This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

Re: PDF to Text, a challenging problem

#23

Post by cpursley »

andrethegiant wrote: post #15

Cloudflare’s ai.toMarkdown() function available in Workers AI can handle PDFs pretty easily. Judging from speed alone, it seems they’re parsing the actual content rather than shoving into OCR/LLM. Shameless plug: I use this under the hood when you prefix any PDF URL with https://pure.md/ to convert to raw text.

How's their function do on complex data tables, charts and that sort of stuff?

Re: PDF to Text, a challenging problem

#24

Post by kapitalx »

bob1029 wrote: post #19

When accommodating the general case, solving PDF-to-text is approximately equivalent to solving JPEG-to-text. The only PDF parsing scenario I would consider putting my name on is scraping AcroForm field values from standardized documents.

This is approximately the approach we're taking also at https://doctly.ai, add to that a "multiple experts" approach for analyzing the image (for our 'ultra' version), and we get really good results. And we're making it better constantly.

Re: PDF to Text, a challenging problem

#25

Post by dwheeler »

The better solution is to embed, in the PDF, the editable source document. This is easily done by LibreOffice. Embedding it takes very little space in general (because it compresses well), and then you have MUCH better information on what the text is and its meaning. It works just fine with existing PDF readers.

Re: PDF to Text, a challenging problem

#27

Post by andrethegiant »

burkaman wrote: post #18
andrethegiant wrote: post #15

Cloudflare’s ai.toMarkdown() function available in Workers AI can handle PDFs pretty easily. Judging from speed alone, it seems they’re parsing the actual content rather than shoving into OCR/LLM. Shameless plug: I use this under the hood when you prefix any PDF URL with https://pure.md/ to convert to raw text.

If you're looking for test cases, this is the first thing I tried and the result is very bad: https://pure.md/https://docs.house.gov/meetings/IF/IF00/2025...

Apart from lacking newlines, how is the result bad? It extracts the text for easy piping into an LLM.

Re: PDF to Text, a challenging problem

#28

Post by svat »

whenc wrote: post #13
svat wrote: post #10

One thing I wish someone would write is something like the browser's developer tools ("inspect elements") for PDF — it would be great to be able to "view source" a PDF's content streams (the BT … ET operators that enclose text, each Tj operator for setting down text in the currently chosen font, etc), to see how every “pixel” of the PDF is being specified/generated. I know this goes against the current trend / state-…

Try with cpdf (disclaimer, wrote it): cpdf -output-json -output-json-parse-content-streams in.pdf -o out.json Then you can play around with the JSON, and turn it back to PDF with cpdf -j out.json -o out.pdf No live back-and-forth though.

The live back-and-forth is the main point of what I'm asking for — I tried your cpdf (thanks for the mention; will add it to my list) and it too doesn't help; all it does is, somewhere 9000-odd lines into the JSON file, turn the part of the content stream corresponding to what I mentioned in the earlier comment into:

        [
          [ { "F": 0.0 }, "g" ],
          [ { "F": 0.0 }, "G" ],
          [ { "F": 0.0 }, "g" ],
          [ { "F": 0.0 }, "G" ],
          [ "BT" ],
          [ "/F19", { "F": 10.9091 }, "Tf" ],
          [ { "F": 88.93600000000001 }, { "F": 709.0410000000001 }, "Td" ],
          [
            [
              "Subsequen",
              { "F": 28.0 },
              "t",
              { "F": -374.0 },
              "to",
              { "F": -373.0 },
              "the",
              { "F": -373.0 },
              "p",
              { "F": -28.0 },
              "erio",
              { "F": -28.0 },
              "d",
              { "F": -373.0 },
              "analyzed",
              { "F": -373.0 },
              "in",
              { "F": -374.0 },
              "our",
              { "F": -373.0 },
              "study",
              { "F": 83.0 },
              ",",
              { "F": -383.0 },
              "Bridge's",
              { "F": -373.0 },
              "paren",
              { "F": 27.0 },
              "t",
              { "F": -373.0 },
              "compan",
              { "F": 28.0 },
              "y",
              { "F": -373.0 },
              "Ne",
              { "F": -1.0 },
              "wGlob",
              { "F": -27.0 },
              "e",
              { "F": -374.0 },
              "reduced"
            ],
            "TJ"
          ],
          [ { "F": -16.936 }, { "F": -21.922 }, "Td" ],
This is just a more verbose restatement of what's in the PDF file; the real questions I'm asking are:

- How can a user get to this part, from viewing the PDF file? (Note that the PDF page objects are not necessarily a flat list; they are often nested at different levels of “kids”.)

- How can a user understand these instructions, and “see” how they correspond to what is visually displayed on the PDF file?

Re: PDF to Text, a challenging problem

#29

Post by layer8 »

bob1029 wrote: post #19

When accommodating the general case, solving PDF-to-text is approximately equivalent to solving JPEG-to-text. The only PDF parsing scenario I would consider putting my name on is scraping AcroForm field values from standardized documents.

If you assume standardized documents, you can impose the use of Tagged PDF: https://pdfa.org/resource/tagged-pdf-q-a/

Re: PDF to Text, a challenging problem

#30

Post by mediaman »

marginalia_nu wrote: post #12
xnx wrote: post #5

Weird that there's no mention of LLMs in this article even though the article is very recent. LLMs haven't solved every OCR/document data extraction problem, but they've dramatically improved the situation.

Author here: LLMs are definitely the new gold standard for smaller bodies of shorter documents. The article is in the context of an internet search engine, the corpus to be converted is of order 1 TB. Running that amount of data through an LLM would be extremely expensive, given the relatively marginal improvement in outcome.

Corpus size doesn't mean much in the context of a PDF, given how variable that can be per page.

I've found Google's Flash to cut my OCR costs by about 95+% compared to traditional commercial offerings that support structured data extraction, and I still get tables, headers, etc from each page. Still not perfect, but per page costs were less than one tenth of a cent per page, and 100 gb collections of PDFs ran to a few hundreds of dollars.

Post reply on HN