Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

121–130 of 350 posts

Re: What's so hard about PDF text extraction?

#121

Post by jiveturkey »

sixhobbits wrote: post #9

As a meta point, it's really nice to see such a well-written, well-researched article that is obviously used as a form of lead generation for the company, and yet no in your face "call to actions" which try to stop you reading the article you came for and get out your wallet instead.

i mean except for the banner at the top and bottom! but yeah, an SEO article with actual substance, well formatted, not grey-on-grey[1], no trackers[2], is rare these days.

[1] recently read an SEO post on okta's site. who can read that garbage?

[2] only GA ... which isn't a 3rd-party tracker.

Re: What's so hard about PDF text extraction?

#122

Post by eterps »

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

Lots of people doing their daily jobs are not aware of the information loss that occurs whenever they are saving/exporting as PDF.

Re: What's so hard about PDF text extraction?

#123

Post by thayne »

I've worked on the other end of this: trying to make it easy to extract text from PDFs that we generated. Turns out that is pretty hard too. There just isn't a good way to include metadata about how text flows. So columns, callouts, captions, etc. all cause problems. The PDF format just wasn't designed for text extraction.

Re: What's so hard about PDF text extraction?

#124

Post by staticautomatic »

jazzido wrote: post #116
saradhi wrote: post #79

I'm an ML engineer, worked as a part time data engineer consultant for a medical lines/claims extraction company, for 3 years, which majorly involved in extracting the tabular data from the PDFs and Images. Developer rules or parsers as such is JUST no help. You end up creating a new rule every time you miss the data extraction. With that in consideration, and the existing resources are little help especially on skew…

Hi, author and maintainer of Tabula ( https://github.com/tabulapdf/tabula ). We've been trying to contact you about the "Tabula Pro" version that you are offering. Feel free to reachme at manuel at jazzido dot com

Edit: See reply below

Am I reading the repos correctly? It looks like Extractable copied Tabula (MIT) to its own repo rather than forking it, removed the attribution, and then tried to re-license it as Apache 2.0. If so, that would be pretty fucked up.

https://github.com/tabulapdf

https://github.com/ExtractTable/tabulapro

Re: What's so hard about PDF text extraction?

#125

Post by serhart »

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

> Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you may want to learn about Chomsky hierarchy of formal languages.

Most programming languages offer a regex engine capable of matching non-regular languages. I agree though, if you are actually trying to _parse_ text then a regex is not the right tool. It just depends on your use case.

Re: What's so hard about PDF text extraction?

#126

Post by mqus »

hwc wrote: post #24

I worked on PDF generating software for years. It's a horrible format that should never have been approved as an ISO standard. When in doubt, use plain text. It's a million times better in every way that counts. I wish my bank statements and such could be downloaded as plain text files, instead of massive PDF files that embed another copy of a bunch of typefaces in each file.

Since gdpr, businesses are "required" to make your data available to you for transfer in a machine-readable format and you could argue that pdf is not exactly machine-readable in the sense of the law. Practically I have seen cases where you do get csv or something similar, but especially smaller firms will probably give you word documents, excel files or pdfs.

Re: What's so hard about PDF text extraction?

#127

Post by staticautomatic »

UglyToad wrote: post #75

The open source project I work on [0] returns the letters, their positions and other associated information. We provide support for retrieving words as well as a bunch of different algorithms for document layout analysis [1]. But like the other commenters here mention, it's an extremely difficult problem which doesn't have an easy or general solution. I was trying to build a custom library on top of the open-source l…

I've also done a lot of work in this space and one thing I don't understand is why more extraction libraries don't support images as input. If your PDF isn't layered or OCR'd, it might as well be an image. I've lost count of the number of times I've downloaded some PDF extraction tool and then had to hack it into accepting an image.

Re: What's so hard about PDF text extraction?

#128

Post by bachmeier »

lm28469 wrote: post #44
Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

PDF is good at what it's supposed to be good. Parsing pdf to extract data is like using a rock as a hammer and a screw as a nail, if you try hard enough it'll eventually work but it was never intended to be used that way.

It's pretty silly when you think about it. There's an underlying assumptions that you'll work with the data in the original format that you used to make the PDF.

Re: What's so hard about PDF text extraction?

#129

Post by shim__ »

LeonM wrote: post #60
slig wrote: post #34

Earlier quoted context omitted.

Maybe there's a SasS opportunity for you to explore.

I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down. The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing. Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice. Off course there were workarounds…

I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.

Re: What's so hard about PDF text extraction?

#130

Post by fluidcruft »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

Do you have any tricks for dealing with missing unicode character mapping tables for embedded fonts?
Post reply on HN