Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

321–330 of 350 posts

Re: What's so hard about PDF text extraction?

#321

Post by jazzido »

wpietri wrote: post #285
saradhi wrote: post #279

Earlier quoted context omitted.

William, the intention of "TabulaPro" is to give the developers a chance to use a single library instead of switching ExtractTable for images and tabula-py for text PDFs. What do you recommend us to do, to not make you feel we made a dick move. TIA

Well, let me ask a few questions: Did you ask permission of the original author to use a derived name? Did you discuss your plan to commercialize the original author's work with the author? Before starting out? Since starting a commercial project, how much money have you given to the original author?

Author here.

1) No. 2) No. 3) Zero.

Re: What's so hard about PDF text extraction?

#322

Post by saradhi »

wpietri wrote: post #285
saradhi wrote: post #279

Earlier quoted context omitted.

William, the intention of "TabulaPro" is to give the developers a chance to use a single library instead of switching ExtractTable for images and tabula-py for text PDFs. What do you recommend us to do, to not make you feel we made a dick move. TIA

Well, let me ask a few questions: Did you ask permission of the original author to use a derived name? Did you discuss your plan to commercialize the original author's work with the author? Before starting out? Since starting a commercial project, how much money have you given to the original author?

- No, No, Zero.

"commercialize the original author's work with the author" - No, but let me highlight this, any extraction with tabula-py is not commercialized - you can look into the wrapper too :) or even compare the results with tabula-py vs tabulaPro.

Copying the TabulaPro description here, "TabulaPro is a layer on tabula-py library to extract tables from Scan PDFs and Images." - we respect every effort of the contributors & author, never intended to plagiarize.

I understand the misinterpretation here is that we are charging for the open-sourced library because of the name. We already informed author in the email about unpublishing the library, this morning, I just deleted the project and came here to mention it is deleted :)

Re: What's so hard about PDF text extraction?

#323

Post by wpietri »

saradhi wrote: post #322
wpietri wrote: post #285

Earlier quoted context omitted.

Well, let me ask a few questions: Did you ask permission of the original author to use a derived name? Did you discuss your plan to commercialize the original author's work with the author? Before starting out? Since starting a commercial project, how much money have you given to the original author?

- No, No, Zero. "commercialize the original author's work with the author" - No, but let me highlight this, any extraction with tabula-py is not commercialized - you can look into the wrapper too :) or even compare the results with tabula-py vs tabulaPro. Copying the TabulaPro description here, "TabulaPro is a layer on tabula-py library to extract tables from Scan PDFs and Images." - we respect every effort of the co…

Sorry, Saradhi, I don't think you can reasonably claim there was no intention to plagiarize. Adding a "pro" to something is clearly meant to suggest it's the paid version of something. And it's equally clear that "TabulaPro" is derived from "Tabula".

It may be that you didn't realize that people would see your appropriation as wrong, although I have a hard time believing that as well given that the author tried to contact you and was ignored. As they say, "The wicked flee when no man pursueth."

So what I see here is somebody knowingly doing something dodgy and then panicking when getting caught. If you'd really like to make amends, I'd start with some serious introspection on what you actually did, and an honest conversation with the original author that hopefully includes a proper [1] apology.

[1] Meaning it includes an explicit recognition of your error and the harms done, a clear expression of regret, and a sincere offer to make amends. E.g., https://greatergood.berkeley.edu/article/item/the_three_part...

Re: What's so hard about PDF text extraction?

#324

Post by ogurechny »

BashiBazouk wrote: post #282

It's interesting to see other views of PDF. As someone who lives in Illustrator ripping every little piece of data out of a pdf to import into an Illustrator or InDesign file and then making a production pdf for large format printing and fixing plenty of issues along the way I find the text almost inconsequential to the whole thing. It's just another element among many elements: images, vector illustrations, et. PDF…

You are probably working directly with native formats embedded in PDF without even processing the visualized elements. Adobe tools like to do that.

Sometimes, publishers make their PDF e-books from printed source in which images are “optimized” to low quality JPEGs, but next to them non-display Photoshop data streams with pristine megapixel illustrations are kept. If you catch big PDF files, check their insides, it's one line of `mupdf extract`.

Re: What's so hard about PDF text extraction?

#326

Post by floriol »

aasasd wrote: post #304
alkonaut wrote: post #295

Earlier quoted context omitted.

But when you access a paper it’s for reading it, correct? It is worrying if places that are “libraries” of knowledge aren’t taking the opportunity to keep searchable/parseable data, but it’s no worse than a library of books.

> but it’s no worse than a library of books That's not my complaint in the first place. The problem is that while we progressed beyond books on the device side in terms of even just the viewport, we seemingly can't move past the letter-sized paged format. The format may be a bit better than books—what with it being easily distributed and with occasionally copyable text—but not enough so. I'm not even touching the top…

Yeah, it's better with HTML than with PDF, but it's still pretty terrible... Use some actually structured data format like XML (XHTML would be good), because you don't want to include a complete browser just to search for text

Re: What's so hard about PDF text extraction?

#327

Post by floriol »

animalnewbie wrote: post #302

I wabt to read the article but it's pointless because PDF doesn't test anything but ASCII chars well. Add some Asian languages and there is no way to get that text back. PDF needs to die. Djvu is good.

What do you mean? As I understand it it depends on the font - you can provide any sort of encoding. So Unicode is there, I don't see how that would be harder than with the latin abc (which is still a hard problem as per the article)

Re: What's so hard about PDF text extraction?

#328

Post by floriol »

martingoodson wrote: post #114

I wouldn't bother with parsing the pdf. Directly reading from pixels can be more accurate than the parsed output, but will require some R&D. You'll need very high recall text detection and an accurate algorithm for OCR. And a lot of real documents as training data. It's critical that the training data is good quality and much of the engineering effort should go into good annotation interfaces. We built an end-to-end…

As he wrote it in the article, OCR is an order of magnitude slower.

Re: What's so hard about PDF text extraction?

#329

Post by dreamcompiler »

Alex3917 wrote: post #179
tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Except the stakes are lower. Nobody dies if a PDF extraction isn't perfect.

Re: What's so hard about PDF text extraction?

#330

Post by dreamcompiler »

tastyminerals wrote: post #290
greycol wrote: post #199

Earlier quoted context omitted.

Almost any business that has physical suppliers or business customers. PDF is de-facto standard for any invoicing, POs, quotes, etc. If you solve the problem you can effectively programmatically deal with invoicing/payments/ large parts of ordering/dispensing. It's a no brainer to add it on to almost any financial/procurement software that deals with inter business stuff. Any small-medium physical business can probab…

Yes, most invoices are in PDF but only about 40% of them are native PDF meaning they are actual documents not scanned images converted to PDFs. There are are also compound PDF invoices which contain images. So, in order to extract data from them, one needs not only good PDF parser but an OCR engine too.

This is a huge pet peeve of mine. Most invoices are generated on a computer (often in Word) but a huge fraction of the people who generate them don't know how to export to a PDF. So they print the invoice on paper, scan it back in to a PDF, and email that to you. Thus the proliferation of bitmap PDFs.
Post reply on HN