Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to parse a PDF?

eliot-jones.com

Post reply on HN
Print view View original

151–160 of 236 posts

Re: So you want to parse a PDF?

#151

Post by xp84 »

mft_ wrote: post #146

I've been pondering for a while that we need to move away from layout-based written communication. As in, the need to make things look professionally laid out is an anachronism, and is (very) rarely related to comprehension of the actual content. For example, submissions to regulatory agencies are huge documents; we spend lots of time in (typically) Microsoft Word creating documents that follow a layout tradition. As…

I’m with you on PDF, but is docx really that bad in practice? I have not implemented a parser for it so I’m not pushing one answer to that. But it seems like it’s an XML-based format that isn’t about absolutely positioning everything unless you explicitly decide to, and intuitively it seems like it should be like an 80 on the parsing easiness scale if a JPEG is a 0, a PDF is a 15, and a markdown is 100.

Re: So you want to parse a PDF?

#152

Post by Hackbraten »

ponooqjoqo wrote: post #46
wackget wrote: post #3

> So you want to parse a PDF? Absolutely not. For the reasons in the article.

Would be nice if my banks provided records in a more digestible format, but until then, I have no choice.

In Germany, traditional banks and credit unions offer a financial API called FinTS [0]. A couple of desktop banking apps support FinTS, and consumers can typically use it free of charge.

The API has been around since 1998 and is one of the best pieces of software ever produced in Germany imho (if we ignore for a second that that bar is pretty low to begin with).

Unfortunately, it’s mostly traditional German banks and credit unions that offer FinTS. From a neobank’s point of view, chances are you’re catering to a global audience, so you just cobble together a questionable smartphone app and call it a day. That’s probably cheaper and makes more sense than offering a protocol that only works in Germany.

I wish FinTS had caught on internationally though!

[0]: https://en.wikipedia.org/wiki/FinTS

Re: So you want to parse a PDF?

#153

Post by Zardoz84 »

xp84 wrote: post #151
mft_ wrote: post #146

I've been pondering for a while that we need to move away from layout-based written communication. As in, the need to make things look professionally laid out is an anachronism, and is (very) rarely related to comprehension of the actual content. For example, submissions to regulatory agencies are huge documents; we spend lots of time in (typically) Microsoft Word creating documents that follow a layout tradition. As…

I’m with you on PDF, but is docx really that bad in practice? I have not implemented a parser for it so I’m not pushing one answer to that. But it seems like it’s an XML-based format that isn’t about absolutely positioning everything unless you explicitly decide to, and intuitively it seems like it should be like an 80 on the parsing easiness scale if a JPEG is a 0, a PDF is a 15, and a markdown is 100.

Docx it's a proprietary format. So it's a direct no

Re: So you want to parse a PDF?

#154

Post by reactordev »

joakleaf wrote: post #125
throwaway4496 wrote: post #73

Earlier quoted context omitted.

I do PDF for a living, millions of PDFs per month, this is complete nonsense. There is no way you get better results from rastering and OCR than rendering into XML or other structured data.

I sort of agree... I do the same. We also parse millions of PDFs per month in all kinds languages (both Western and Asian alphabets). Getting the basics of PDF parsing to work is really not that complicated -- A few months work. And is an order of magnitude more efficient than generating an image in 300-600 DPI and doing OCR or Visual LLM. But some of the challenges (which we have solved) are: • Glyphs to unicode tab…

We tried the xml structured route, only to end up with pea soup afterwards. Rasterizing and OCR was the only way to get standardized output.

Re: So you want to parse a PDF?

#155

Post by Anon_troll »

xp84 wrote: post #151
mft_ wrote: post #146

I've been pondering for a while that we need to move away from layout-based written communication. As in, the need to make things look professionally laid out is an anachronism, and is (very) rarely related to comprehension of the actual content. For example, submissions to regulatory agencies are huge documents; we spend lots of time in (typically) Microsoft Word creating documents that follow a layout tradition. As…

I’m with you on PDF, but is docx really that bad in practice? I have not implemented a parser for it so I’m not pushing one answer to that. But it seems like it’s an XML-based format that isn’t about absolutely positioning everything unless you explicitly decide to, and intuitively it seems like it should be like an 80 on the parsing easiness scale if a JPEG is a 0, a PDF is a 15, and a markdown is 100.

Extracting text from DOCX is easy. Anything related to layout is non-trivial and extremely brittle.

To get the layout correct, you need to reverse engineer details down to Word's numerical accuracy so that content appears at the correct position in more complex cases. People like creating brittle documents where a pixel of difference can break the layout and cause content to misalign and appear on separate pages.

This will be a major problem for cases like the text saying "look at the above picture" but the picture was not anchored properly and floated to the next page due to rendering differences compared to a specific version of Word.

Re: So you want to parse a PDF?

#156

Post by grues-dinner »

xp84 wrote: post #151
mft_ wrote: post #146

I've been pondering for a while that we need to move away from layout-based written communication. As in, the need to make things look professionally laid out is an anachronism, and is (very) rarely related to comprehension of the actual content. For example, submissions to regulatory agencies are huge documents; we spend lots of time in (typically) Microsoft Word creating documents that follow a layout tradition. As…

I’m with you on PDF, but is docx really that bad in practice? I have not implemented a parser for it so I’m not pushing one answer to that. But it seems like it’s an XML-based format that isn’t about absolutely positioning everything unless you explicitly decide to, and intuitively it seems like it should be like an 80 on the parsing easiness scale if a JPEG is a 0, a PDF is a 15, and a markdown is 100.

The docx standard, which was rather tendentiously named Office Open XML back when OpenOffice was still called that, is 5000 page long and that's only Part 1 of ECMA-376, with another 1500 pages of "Transitional OOXML" in Part 4 which is basically Word-specific quirks.

Re: So you want to parse a PDF?

#157

Post by beng-nl »

jupin wrote: post #124

> Assuming everything is well behaved and you have a reasonable parser for PDF objects this is fairly simple. But you cannot assume everything is well behaved. That would be very foolish, foolish indeed. You're in PDF hell now. PDF isn't a specification, it's a social construct, it's a vibe. The more you struggle the deeper you sink. You live in the bog now, with the rest of us, far from the sight of God. This put a…

Could’ve been written by the great James Mickens.

Re: So you want to parse a PDF?

#158

Post by pbronez »

actionfromafar wrote: post #114
vander_elst wrote: post #81

Earlier quoted context omitted.

It might sound absurd, but on paper this should be the best way to approach the problem. My understanding is that PDFs are intended to produce an output that is consumed by humans and not by computers, the format seems to be focused on how to display some data so that a human can (hopefully) easily read them. Here it seems that we are using a technique that mimics the human approach, which would seem to make sense. I…

1. It's extra work to add an annotation or "internal data format" inside the PDF. 2. By the time the PDF is generated in a real system, the original data source and meaning may be very far off in the data pipeline. It may require incredible cross team and/or cross vendor cooperation. 3. Chicken and egg. There are very few if any machine parseable PDFs out there, so there is little demand for such. I'm actually much m…

That is a really interesting idea. Did some napkin math:

Consumer printers can reliably handle 300 Dots Per Inch (DPI). Standard letter paper is 8.5” x 11” and we need a 0.5” margins on all sides to be safe. This gives you a 7.5” x 10” printable area, which is 2250 x 3000 Dots. Assume 1 Dot = 1 QR Code module (cell) and we can pack 432 Version 26 QR codes onto the page (121 modules per side; 4 modules quiet space buffer between them).

A version 26 QR code can store 864 to 1,990 alphanumeric characters depending on error correction level. That’s 373,248 to 859,680 characters per page! Probably need maximum error correction to have any chance of this working.

If we use 4 dots per module, we drop down to 48 Version 18 QR codes (6 x 8). Those can hold 452-1046 alphanumeric characters each, for 20,000 - 50,208 characters per page.

Compare that at around 5000 characters per page of typed English. You can conservatively get 4x the information density with QR codes.

Conclusion: you can add a machine-readable appendix to your text-only PDF file at a cost of increasing page count by about 25%.

Re: So you want to parse a PDF?

#159

Post by MOARDONGZPLZ »

MartinMond wrote: post #148
hobs wrote: post #139

Earlier quoted context omitted.

Looks super interesting, except for there's no pricing on the page that I could find except for contact sales - totally understand wanting to do a higher touch sales process, but that's going to bounce some % of eng types who want to try things out but have been bamboozled before.

I know - we're working on adding Self-Serve sign up for non-enterprise deals. But regarding our pricing - I can point you at an actual testimonial https://www.g2.com/products/pspdfkit-sdk/reviews/pspdfkit-sd... > These pricing structures can be complex and NEED to be understood fully before moving forward with purchase. However, out of all of the solutions that I reviewed, [Nutrient] was the one that walked me throug…

I love that the employee’s (CEO’s?) response to a “there’s no pricing on your website” comment is a link to a review on another kinda random website of a testimonial that getting pricing from them sucks and was marginally above the baseline of “the customer didn’t get scammed.” Ringing endorsement, along with the implied “we’ve been doing this ten years and still haven’t been able to implement self service sign up or even an html pricing page on the site.”

Re: So you want to parse a PDF?

#160

Post by Macha »

cylemons wrote: post #78
jiveturkey wrote: post #62

Earlier quoted context omitted.

Doesn't rendering to an image require proper parsing of the PDF?

PDF is more like a glorified svg format than a word format. It only contains info on how the document should look but so semantic information like sentences, paragraphs, etc. Just a bag of characters positioned in certain places.

Sometimes the characters aren’t even characters, just paths
Post reply on HN