Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

191–200 of 236 posts

Re: So you want to parse a PDF?

#191

Post by DannyBee »

throwaway4496 wrote: post #71
do_not_redeem wrote: post #28

Earlier quoted context omitted.

PDFs don't always lay out characters in sequence, sometimes they have absolutely positioned individual characters instead. PDFs don't always use UTF-8, sometimes they assign random-seeming numbers to individual glyphs (this is common if unused glyphs are stripped from an embedded font, for example) etc etc

But all those problems exist when rendering into a surface or rastering. I just don't understand how one thinks, this is a hard problem, let me make it harder by solving the problem into another kind of problem that is just as hard as solving it in the first place (PDF to structured data vs PDF to raster). And then solve the new problem, which is also hard. It is absurd.

The problems don't actually exist in the way you think.

When extracting text directly, the goal is to put it back into content order, regardless of stream order. Then turn that into a string. As fast as possible.

That's straight text. if you want layout info, it does more. But it's also not just processing it as a straight stream and rasterizing the result. It's trying to avoid doing that work.

This is non-trivial on lots of pdfs, and a source of lots of parsing issues/errors because it's not just processing it all and rasterizing it, but trying to avoid doing that.

When rasterizing, you don't care about any of this at all. PDFs were made to raster easily. It does not matter what order the text is in the file, or where the tables are, because if you parse it straight through, raster, and splat it to the screen, it will be in the proper display order and look right.

So if you splat it onto the screen, and then extract it, it will be in the proper content/display order for you. Same is true of the tables, etc.

So the direct extraction problems don't exist if you can parse the screen into whatever you want, with 100% accuracy (and of course it doesn't matter if you use AI or not to do it).

Now, i am not sure i would use this method anyway, but your claim that the same problems exist is definitely wrong.

Re: So you want to parse a PDF?

#193

Post by UglyToad »

Animats wrote: post #55

Can you just ignore the index and read the entire file to find all the objects?

Yes this is generally the fallback approach if finding the objects via the index (xref) fails. It is slightly slower but it's a one time cost, though I imagine it was a lot slower back when PDFs were first used on the machines of the time.

Re: So you want to parse a PDF?

#194

Post by phaistra »

mft_ wrote: post #146

I've been pondering for a while that we need to move away from layout-based written communication. As in, the need to make things look professionally laid out is an anachronism, and is (very) rarely related to comprehension of the actual content. For example, submissions to regulatory agencies are huge documents; we spend lots of time in (typically) Microsoft Word creating documents that follow a layout tradition. As…

Sounds like you are describing markdown.

Re: So you want to parse a PDF?

#195

Post by Kranar »

pavel_lishin wrote: post #186
peterfirefly wrote: post #184

Earlier quoted context omitted.

Your Geoff problem could be solved easily by not putting the ligature into the PDF in the first place. You don't need the cooperation of the entire rest of the world (at the cost of hundreds of millions of dollars) to solve that one little problem that is at most a tiny inconvenience.

That's right, and all the Günters, Renées and Þórunns out there can just change their names to Gunter, Renee and Thorunn.

None of those names have a ligature. Infact Renées is a "deligatured" spelling of Renæes which would be incredibly rare.

Re: So you want to parse a PDF?

#196

Post by ponooqjoqo »

vander_elst wrote: post #83
ponooqjoqo wrote: post #46

Earlier quoted context omitted.

Would be nice if my banks provided records in a more digestible format, but until then, I have no choice.

I find it pretty sad that for some banks the CSV export is behind a paywall.

Mine offer CSV exports but the data in the CSV file is a small fraction of the data in the PDF statement.

It's just a list of transactions, not a reconcilable "end of month" balance with all the data.

Re: So you want to parse a PDF?

#197

Post by sidebute »

throwaway4496 wrote: post #79
sidebute wrote: post #32

Earlier quoted context omitted.

> Sounds like "I don't know programming, so I will just use AI". If you were leading Tensorlake, running on early stage VC with only 10 employees ( https://pitchbook.com/profiles/company/594250-75 ), you'd focus all your resources on shipping products quickly, iterating over unseen customer needs that could make the business skyrocket, and making your customers so happy that they tell everyone and buy lots more licen…

I would hire someone who understands PDFs instead of doing the equivalent of printing a digital document and scanning it for "digital record keeping". Stop everything and hire someone who understands the basics of data processing and some PDF.

What's the economic justification?

Let's assume we have a staff of 10 and they're fully allocated to committed features and deadlines, so they can't be shifted elsewhere. You're the CTO and you ask the BOD for another $150k/y (fully burdened) + equity to hire a new developer with PDF skills.

The COB asks you directly: "You can get a battle-tested PDF parser off-the-shelf for little or no cost. We're not in the PDF parser business, and we know that building a robust PDF parser is an open-ended project, because real-world PDFs are so gross inside. Why are you asking for new money to build our own PDF parser? What's your economic argument?"

And the killer question comes next: "Why aren't you spending that $150k/y on building functionality that our customers need?" If don't give a convincing business justification, you're shoved out the door because, as a CTO, your job is building technology that satisfies the business objectives.

So CTO, what's your economic answer?

Re: So you want to parse a PDF?

#198

Post by sidebute »

DannyBee wrote: post #187
sidebute wrote: post #36

Earlier quoted context omitted.

While we have a PDF internals expert here, I'm itching to ask: Why is mupdf-gl so much faster than everything else? (on vanilla desktop linux) Its search speed on big pdfs is dramatically faster than everything else I've tried and I've often wondered why the others can't be as fast as mupdf-gl. Thanks for any insights!

It's funny you ask this - i have spent a time building pdf indexing/search apps on the side over the past few weeks. I'll give you the rundown. The answer to your specific question is basically "some of them process letter by letter to put text back in order, and some don't. Some build fast trie/etc based indexes to do searching, some don't" All of my machine manuals/etc are in PDF, and too many search apps/OS search…

> From what i can tell digging into the code that's available, ..., how much time they take to put the text back in content order... The slowest are doing letter by letter. The fastest are not.

Thank you, that's really helpful.

I hadn't considered content reordering but it makes perfect sense given that the internal character ordering can be anything, as long as the page renders correctly. There's an interesting comp-sci homework project: Given a document represented by an unordered list of tuples [ (pageNum, x, y, char) ], quickly determine whether the doc contains a given search string.

Sometimes I need to search PDFs for a regex and use pdfgrep. That builds on poppler/xpdf, which extracts text >2x slower than mupdf (https://documentation.help/pymupdf/app1.html#part-2-text-ext..., fitz vs xpdf). From this discussion, I'm now writing my own pdfgrep that builds on mupdf.

Re: So you want to parse a PDF?

#200

Post by acuozzo »

pjc50 wrote: post #131
gcanyon wrote: post #57

The answer seems obvious to me: 1. PDFs support arbitrary attached/included metadata in whatever format you like. 2. So everything that produces PDFs should attach the same information in a machine-friendly format. 3. Then everyone who wants to "parse" the PDF can refer to the metadata instead. From a practical standpoint: my first name is Geoff. Half the resume parsers out there interpret my name as "Geo" and "ff" s…

"Should" is doing a lot of heavy lifting here. I think people underestimate how much use of PDF is actually adversarial; starting with using it for CVs to discourage it being edited by middlemen, then "redaction" by drawing boxes over part of the image, encoding tables in PDF rather than providing CSV to discourage analysis, and so on.

> starting with using it for CVs to discourage it being edited by middlemen

Isn't the motivation to convey that you care enough about your CV to care about its typesetting?

I've seen .docx CVs get so trashed (metadata loss?) that they looked like they were typeset by a sloppy/uncaring person or a child.

Post reply on HN