Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

101–110 of 350 posts

Re: What's so hard about PDF text extraction?

#101

Post by nradov »

fermienrico wrote: post #93
aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

Actually, no thanks. "Sementic" structure is how we got responsive web soup of ugly websites with hamburger menus. We need the opposite, we need a format that stays the same size, same proportions and is vectorized so you can zoom to any size - however, the relationship of space between elements remains constant. PDF is an amazing format IMO. Think of it like Docker - the designer knows exactly how its going to appea…

The problems you describe have nothing to do with the semantic web. Those are orthogonal issues.

Re: What's so hard about PDF text extraction?

#102

Post by goatlover »

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

It was never meant to be a data storage format. It's for reading and printing.

Re: What's so hard about PDF text extraction?

#103

Post by ProZsolt »

mumblemumble wrote: post #62
ProZsolt wrote: post #59

Earlier quoted context omitted.

PDF is great what it meant to be, a digital printed paper, with its pros (It will look exactly the same anywhere) and cons (Can't easily extract data from it or modify it). Currently, there is no viable alternative if you want the pros but not the cons

> Currently, there is no viable alternative if you want the pros but not the cons I remember OpenXPS being much easier to work with. That might be due to cultural rather than structural differences, mind - fewer applications generate OpenXPS, so there's fewer applications to generate them in their own special snowflake ways.

This is the first time I heard of it. When I search for it I only find the Wikipedia article and 99 links to how to convert it to pdf.

The problem with this is that from an average person perspective it doesn't have the pros. There is no built-in or first-party app that can open this format on Mac and Linux. More than 99% of the users only want to read or print it. It's hard to convince them to use an alternative format when it's way more difficult to do the only thing they want to do.

Re: What's so hard about PDF text extraction?

#104

Post by [deleted] »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

[deleted]

Re: What's so hard about PDF text extraction?

#105

Post by saradhi »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Interestingly, I was doing the similar stuff for 3 years to a US company. Curious, is your client a legal tech company? Mine was.

The experience helped me to roll out an API, as https://extracttable.com, for developers.

OCR tricks? Assuming post processing dev stuff - may I know your OCR engine. We are supported with Kofax and openText along with cloud engines like GVision as a backup.

Re: What's so hard about PDF text extraction?

#106

Post by CaptArmchair »

saradhi wrote: post #79

I'm an ML engineer, worked as a part time data engineer consultant for a medical lines/claims extraction company, for 3 years, which majorly involved in extracting the tabular data from the PDFs and Images. Developer rules or parsers as such is JUST no help. You end up creating a new rule every time you miss the data extraction. With that in consideration, and the existing resources are little help especially on skew…

I chuckled at your "the worst image" sample. Which still looked quite decent all things considered.

You're "handwritten" example looks a bit "too decent" as well. I can see how that works. You first look for the edges of the table, and then you evaluate the symbol in each cell as something that matches unicode.

So, how well does this cope with increasing degradation? i.e. pencil written notes that bleed outside cell borders, curve around borders, etc.? Stamps and symbols (watermarks) across tables?

Re: What's so hard about PDF text extraction?

#107

Post by robinhowlett »

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

Thanks for the links - agree about the (x,y,text) callout but other metadata like font size can be useful too.

Regexes have limitations but I was able them to leverage them sufficiently for PDFs from a single source.

I parsed over 1 million PDFs that had a fairly complex layout using Apache PDFBox and wrote about it here: https://www.robinhowlett.com/blog/2019/11/29/parsing-structu...

Re: What's so hard about PDF text extraction?

#108

Post by giovannibonetti »

robinhowlett wrote: post #107
giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

Thanks for the links - agree about the (x,y,text) callout but other metadata like font size can be useful too. Regexes have limitations but I was able them to leverage them sufficiently for PDFs from a single source. I parsed over 1 million PDFs that had a fairly complex layout using Apache PDFBox and wrote about it here: https://www.robinhowlett.com/blog/2019/11/29/parsing-structu...

Oh, yeah, pdf2json returns font sizes as well. I forgot to mention that.

Re: What's so hard about PDF text extraction?

#109

Post by anodyne33 »

Does anybody regularly use Acrobat's text extraction engine? I've had fine results as far as accuracy goes when compared to other OCR engines but one sticking point drives me nuts. My problem is, and I'm typically doing this in batches of thousands of files, if a PDF has a footer applied Acrobat sees that as renderable text and blows off the rest of the rest of the page. I've tried all manner of sanitizing, removing hidden information, saving as another PDF protocol and still can't get around the plain text footers/headers. In a perfect world I'd have unlimited Tesseract or ABBYY access but we're trying to do this on the cheap and I'm working with client data that I don't want to bang through Google. I'll have to poke at some of the open source tools mentioned so far, too.

Re: What's so hard about PDF text extraction?

#110

Post by daniel-levin »

teddyc wrote: post #61
daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Can I PM you?

Of course.
Post reply on HN