What's so hard about PDF text extraction? • Hacker News Board

Re: What's so hard about PDF text extraction?

#171

Post by aasasd » Tue, Mar 03, 2020, 7:30 PM UTC

machinelabo wrote: post #112

nradov wrote: post #101

Earlier quoted context omitted.

The problems you describe have nothing to do with the semantic web. Those are orthogonal issues.

No, not everything is orthogonal. When you have sementic structure, you're gonna display it in responsive and adaptive way. That __breaks__ the design intent. For a designer, WYSIWYG is godsent. the parent comment is right - PDF is like a Docker container for designers - people who work with media. If you have opposing thoughts, please elaborate further instead of simply saying "nothing to do with it". HN works by ex…

> For a designer, WYSIWYG is godsent. the parent comment is right - PDF is like a Docker container for designers - people who work with media.

Sorta like this then, more rigidly conveying the designer's intended layout: https://bureau.rocks/projects/book-typography-en/ ?

(Though these ‘books’ are almost the opposite of what I'm advocating for, in terms of formatting.)

Re: What's so hard about PDF text extraction?

#172

Post by darknoon » Tue, Mar 03, 2020, 7:30 PM UTC

ldenoue wrote: post #50

Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

That’s the approach I’m using to reformat “reflow” PDFs for mobile in my app https://readerview.app/

The first link on your demo gives me an error (mobile safari) https://www.appblit.com/pdfreflow/viewdoc?url=http://arxiv.o...

Re: What's so hard about PDF text extraction?

#173

Post by mark-r » Tue, Mar 03, 2020, 7:31 PM UTC

Not the only format Adobe made overly complex.

Wonderfully amusing code comment, about Adobe PSD format: https://fallenpegasus.livejournal.com/854615.html

Re: What's so hard about PDF text extraction?

#174

Post by bob1029 » Tue, Mar 03, 2020, 7:35 PM UTC

equasar wrote: post #157

bob1029 wrote: post #69

Earlier quoted context omitted.

We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…

Any recommended library for .NET to extract text by coordinates?

I don't know that this could exist for all PDFs.

Sounds like you are in need of OCR if you want to be able to use arbitrary screen coords as a lookup constraint.

Re: What's so hard about PDF text extraction?

#175

Post by KineticLensman » Tue, Mar 03, 2020, 7:36 PM UTC

totololo wrote: post #147

eterps wrote: post #122

Earlier quoted context omitted.

Lots of people doing their daily jobs are not aware of the information loss that occurs whenever they are saving/exporting as PDF.

In the consulting industry I’ve seen PDF being used precisely because third parties couldn’t mess with the content anymore.

Yes, the company I once worked for used to supply locked PDF copies to make it slightly harder for casual readers to re-use / steal our text.

Re: What's so hard about PDF text extraction?

#176

Post by faeyanpiraat » Tue, Mar 03, 2020, 7:41 PM UTC

mLuby wrote: post #158

shim__ wrote: post #129

Earlier quoted context omitted.

I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.

I blame CSS.

Why?

Re: What's so hard about PDF text extraction?

#177

Post by pierre » Tue, Mar 03, 2020, 7:43 PM UTC

That's one of the reason we build Http://par.sr to try to get as clean data of possible from pdf !

Re: What's so hard about PDF text extraction?

#178

Post by gmiller123456 » Tue, Mar 03, 2020, 7:44 PM UTC

pqdfdsaff wrote: post #48

jrandm wrote: post #27

Earlier quoted context omitted.

> Finding an algorithm that approximates how a human approaches a page layout doesn’t feel like it would be all that hard. "In CS, it can be hard to explain the difference between the easy and the virtually impossible." https://xkcd.com/1425/

To be fair, that particular comic's 5 year task was solved shortly after it was posted, at least in part to prove it was wrong.

Without having heard of or tested the solution, I'll bet anyone $1M that I can produce an image that produces an incorrect answer. Which would mean it's not "solved".

Re: What's so hard about PDF text extraction?

#179

Post by Alex3917 » Tue, Mar 03, 2020, 7:47 PM UTC

tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be.

The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as much profit to any company that can solve it. But there are hundreds of billions of dollars going into self-driving cars, and like zero dollars going into this problem.

Re: What's so hard about PDF text extraction?

#180

Post by aasasd » Tue, Mar 03, 2020, 7:47 PM UTC

mLuby wrote: post #165

aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

Not everything needs to look good on every screen size. I don't expect to be able to read academic papers on my smartwatch, and a simple alarm clock app looks kind of silly when it's fullscreen across a desktop monitor. Likewise, when layout makes the difference between reader understanding or confusion, it's hard to trust automatic reflowing on unknown screen sizes. PDF is simply better than HTML when it comes to pr…

I wonder if you realize that both your points wildly miss what I said.

First, there's no need to stretch my argument to the point of it being ridiculous. I don't have to reach for a watch to suffer from PDF. Even a tablet is enough: I don't see many 14" tablets flying off the shelves. I also know for sure that the vast majority of ubiquitous communicator devices, aka smartphones, are about the same size as mine, so everyone with those is guaranteed to have the same shitty experience with papers on their communicators and will have to sedentary-lifestyle their ass off in front of a big display for barely any reason.

Secondly:

> Likewise, when layout makes the difference between reader understanding or confusion, it's hard to trust automatic reflowing on unknown screen sizes. PDF is simply better than HTML when it comes to preserving layout as the author intended.

As I wrote right there above, still no explanation of why the layout makes that difference and why preserving it is so important when most papers are just walls of text + images + some formulas. Somehow I'm able to read those very things off HTML pages just fine.