Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PDF to Text, a challenging problem

marginalia.nu

Post reply on HN
Print view View original

181–190 of 206 posts

Re: PDF to Text, a challenging problem

#181

Post by spacecaps »

bartread wrote: post #3

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a " (anti-)pattern, and especially on sites that use more semantically useful elements, like . Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are s…

I was irritated that I couldn't extract data from PDFs in a similar way to web pages + BeautifulSoup, so I built a library that (kind of) does just that[0]. It does a bunch of other nonsense, but the main goal is a more "human" way of interacting, e.g. `page.find('text:bold:contains("Summary").below().extract_text()`.

And since every PDF is its own bespoke nightmare, I'm also trying to build up a collection of awful-to-extract-data-from examples to serve as the foundation for a how-to library[1].

[0] https://jsoma.github.io/natural-pdf/

[1] https://badpdfs.com/

Re: PDF to Text, a challenging problem

#182

Post by GuB-42 »

hallman76 wrote: post #136
90s_dev wrote: post #63

Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…

We will never get back the collective man-decades of time that has been burned by this format. When will the madness stop?

PDF is effectively digital paper, and it works really well for this. When I made PDFs 20 years ago, I knew they will always look the same on every device, including on paper, and they did, and they still do. In addition, a document is a single file, reasonably compact, looks good on any resolution, and is generally searchable. Even if not ideal, it can also support scans of paper documents in a way that can be sent to a printer on the other side of the planet and you will get the same result as if you had used a copier.

Data extraction is hard, but that's not what it is designed for, it is for people to read, like paper documents.

Far from being "mad", it is remarkably stable. It has some crazy features, and it is not designed for data extraction (but doesn't actively prevent it!). But look at the alternative. Word documents? Html? Svg? One of the zillion XML-based document formats? Markdown? Is any one of these suitable for writing, say, a scientific paper (with maths, tables, graphics...) in a way that is readable by a human on a computer or in print and will still be in decades and that is easier to process by a machine than a PDF?

Re: PDF to Text, a challenging problem

#183

Post by varunneal »

dstryr wrote: post #119
varunneal wrote: post #71

Earlier quoted context omitted.

I've been hacking away at trying to process PDFs into Markdown, having encountered similar obstacles to OP regarding header detection (and many other issues). OCR is fantastic these days but maintaining a global structure to the document is much trickier. Consistent HTML seems still out of reach for large documents. I'm having half-decent results with Markdown using multiple passes of an LLM to extract document struc…

Give this project a try. I've been using it with promising results. https://github.com/matthsena/AlcheMark

I'll check it out!

Re: PDF to Text, a challenging problem

#184

Post by 90s_dev »

selcuka wrote: post #155
90s_dev wrote: post #63

Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…

A while ago someone asked me a C++ question and I said "sorry, never worked seriously with C++". Then I remembered that I wrote the client code for a private instant messenger in Borland C++ approximately 20 years ago, that was used by thousands of people. So yeah, it happens.

Exactly. Every time I look at Rust, it looks foreign and unfamiliar, until I remember that I wrote Rust for a client for maybe a year straight many years ago. Then it starts to come back to me.

Re: PDF to Text, a challenging problem

#185

Post by drguthals »

svat wrote: post #10

One thing I wish someone would write is something like the browser's developer tools ("inspect elements") for PDF — it would be great to be able to "view source" a PDF's content streams (the BT … ET operators that enclose text, each Tj operator for setting down text in the currently chosen font, etc), to see how every “pixel” of the PDF is being specified/generated. I know this goes against the current trend / state-…

"I know this goes against the current trend / state-of-the-art of using vision models to basically “see” the PDF like a human and “read” the text, but it would be really nice to be able to actually understand what a PDF file contains."

Some combination of this is what we're building at Tensorlake (full disclosure I work there). Where you can "see" the PDF like a human and "understand" the contents, not JUST "read" the text. Because the contents of PDFs are usually in tables, images, text, formulas, hand-writing.

Being able to then "understand what a PDF file contains" is important (I think) for that understand part though. And so then we parse the PDF and run multiple models to extract markdown chunks/JSON so that you can ingest the actual data into other applications (AI agents, LLMs, or frankly whatever you want).

https://tensorlake.ai

Re: PDF to Text, a challenging problem

#186

Post by drguthals »

PDF -> Useful Information is what Tensorlake does (https://tensorlake.ai)

Because PDFs are so dominate and yet each one has information in more than just text (tables, images, formulas, hand-writing, strike-throughs even), we (as devs) need to be able have tools that understand the contents, not just "read" them.

Full disclosure...I work there

Re: PDF to Text, a challenging problem

#187

Post by me-vs-cat »

theamk wrote: post #146
protocolture wrote: post #144

Earlier quoted context omitted.

Behold a Bitmap. But for real, thats a pretty easy set of hurdles. Really the barrier is the psychological fallacy that PDF's are immutable.

Should have added "looks good on screen and on paper", "stores text compactly" and "multiple pages supported" :) And yes, that's a pretty easy set of hurdles. I wish we'd standardized on DjVu instead. Re "PDF's are immutable." - that's not a psychological fallacy, that's a primary advantage of PDFs. If I wanted mutable format, I'd take an odt (or rtf or a doc). "Output only" format allows one to use the very latest v…

PDFs are not immutable.

Re: PDF to Text, a challenging problem

#188

Post by me-vs-cat »

harshreality wrote: post #168
theamk wrote: post #146

Earlier quoted context omitted.

Should have added "looks good on screen and on paper", "stores text compactly" and "multiple pages supported" :) And yes, that's a pretty easy set of hurdles. I wish we'd standardized on DjVu instead. Re "PDF's are immutable." - that's not a psychological fallacy, that's a primary advantage of PDFs. If I wanted mutable format, I'd take an odt (or rtf or a doc). "Output only" format allows one to use the very latest v…

What's immutable, without tools to decompress and possibly perform further de-obfuscation of text streams, is the typical way publishing software encodes text into streams inside PDFs. It remains possible to have a pdf with text that is easily mutable with any text editor. Even if text inside a pdf is annoyingly encoded, you can always just replace the appropriate object/text streams... if you can identify the right…

> you can always just replace the appropriate object/text streams

Or right-click and select Edit. Works in several PDF editors, on both text and image content.

Re: PDF to Text, a challenging problem

#189

Post by chaps »

mattigames wrote: post #106
chaps wrote: post #80

Earlier quoted context omitted.

Microsoft is one of the bigger contributors to this. Like -- why does excel have a feature to export to PDF, but not a feature to do the opposite? That export functionality really feels like it was given to a summer intern who finished it in two weeks and never had to deal with it ever again.

Because then we would have 2 formats: "pdfs generated by Excel" and "real pdfs" with the same extension and that would be it's own can of worms for Microsoft's and for everyone else.

Hah, no. We would be going from 200,000 formats to 200,001 formats. Begone, shallow xkcd references!

Re: PDF to Text, a challenging problem

#190

Post by currency »

majora2007 wrote: post #180
theamk wrote: post #137

Earlier quoted context omitted.

When we get an alternative that can: (1) be stored in a single file (2) Allow tables, images and anything else that can be shown on a piece paper (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper (4) won't require Javascript or access to external sites that means never.. We've got lucky we at least got PDF before "web designers" made (3) impossible, and marketers made (4…

Why can't this be done with epub? Single file, all files are packed within the zip, no javascript needed but can be included. Allows for markup and forms, just like pdf.

EPub is, like html, reformattable by the reader, so documents aren't fixed in the way PDFs are.
Post reply on HN