Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

301–310 of 350 posts

Re: What's so hard about PDF text extraction?

#301

Post by speedplane »

aidos wrote: post #193
Alex3917 wrote: post #179

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases. PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane becaus…

> PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned.

Can't stress this enough. The next time you open a multi-column PDF in adobe reader and it selects a set of lines or a paragraph in the way you would expect, know that there is a huge amount of technology going on behind the scenes trying to figure out the start and end of each line and paragraph.

Re: What's so hard about PDF text extraction?

#303

Post by speedplane »

tastyminerals wrote: post #300
thaumasiotes wrote: post #292

Earlier quoted context omitted.

If you're using an OCR engine to understand PDFs that are nothing but a scanned image embedded in a PDF... what do you need a PDF parser for? You can always just render an image of a document and then use that.

For accuracy and speed. The market SOTA Abbyy is far from being accurate.

> The market SOTA Abbyy is far from being accurate.

While Abbyy is likely the best, it's also incredibly expensive. Roughly on the order of $0.01/page or maybe at best a tenth of that in high volume.

For comparison, I run a bunch of OCR servers using the open source tesseract library. The machine-time on one of the major cloud providers works out to roughly $0.01 for 100-1000 pages.

Re: What's so hard about PDF text extraction?

#304

Post by aasasd »

alkonaut wrote: post #295
aasasd wrote: post #249

Earlier quoted context omitted.

> A PDF isn’t for storage it’s for display. It’s the equivalent of a printout. This conjecture would have some practical relevance if I had access to the same papers in other formats, preferably HTML. Yet I'm saddened time and again to find that I don't. In fact, producing HTML or PDF from the same source was exactly my proposed route before I was told that apparently Tex is only good for printing or PDFs. I hope tha…

But when you access a paper it’s for reading it, correct? It is worrying if places that are “libraries” of knowledge aren’t taking the opportunity to keep searchable/parseable data, but it’s no worse than a library of books.

> but it’s no worse than a library of books

That's not my complaint in the first place. The problem is that while we progressed beyond books on the device side in terms of even just the viewport, we seemingly can't move past the letter-sized paged format. The format may be a bit better than books—what with it being easily distributed and with occasionally copyable text—but not enough so.

I'm not even touching the topic of info extraction here, since it's pretty hard on its own and despite it also being better with HTML.

Re: What's so hard about PDF text extraction?

#305

Post by [deleted] »

aasasd wrote: post #289
dang wrote: post #273

Earlier quoted context omitted.

Please don't cross into personal attack in HN comments. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html

By the way, the ‘before’ one also doesn't classify as a personal attack for me. Not sure what your definition of a personal attack is.

[deleted]

Re: What's so hard about PDF text extraction?

#306

Post by minerals29 »

speedplane wrote: post #303
tastyminerals wrote: post #300

Earlier quoted context omitted.

For accuracy and speed. The market SOTA Abbyy is far from being accurate.

> The market SOTA Abbyy is far from being accurate. While Abbyy is likely the best, it's also incredibly expensive. Roughly on the order of $0.01/page or maybe at best a tenth of that in high volume. For comparison, I run a bunch of OCR servers using the open source tesseract library. The machine-time on one of the major cloud providers works out to roughly $0.01 for 100-1000 pages.

It is the best and this is one of the reasons why PDF extraction is hard :)

Re: What's so hard about PDF text extraction?

#307

Post by dang »

aasasd wrote: post #284
dang wrote: post #273

Earlier quoted context omitted.

Please don't cross into personal attack in HN comments. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html

It's pretty obviously not a personal attack.

I read "I think the problem is in your head" as talking about the other user personally. Looking more closely, I can read it as a general statement, in which case it wasn't a personal attack.

Statements of the form "So you're saying [obviously stupid thing]?" still break the site guidelines, though.

https://news.ycombinator.com/newsguidelines.html

Re: What's so hard about PDF text extraction?

#309

Post by kabacha »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

My first internship was at a small company that did PDF parsing and building for EU government agencies and it was really painful work but paid an absolute shitton.

Re: What's so hard about PDF text extraction?

#310

Post by minerals29 »

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

I am an ML engineer in one of the PDF extraction companies processing thousands of invoices and receipts per day in realtime. Before we started adding ML all our processing logic was build on top of hundreds of regexes and gazetteers. Even until now handcrafted rules are the backbone of our extraction system whereas ML is used as fallback. Yes, regexes accumulate tech debt and become a maintenance blackhole but if they work, they are faster and more accurate than any fancy DL tech out there.
Post reply on HN