Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

241–250 of 350 posts

Re: What's so hard about PDF text extraction?

#241

Post by rbobby »

tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> double-tap to zoom

Why wouldn't you just zoom with the center point being where the tap occurred?

Re: What's so hard about PDF text extraction?

#242

Post by appleiigs »

siftrics wrote: post #211
ccozan wrote: post #207

Earlier quoted context omitted.

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

Is your product only on the cloud? My privacy/internet security team won't let me use products that save customer or vendor data on the cloud because you might get hacked. Only giants, like Microsoft, have been approved after an evaluation.

Re: What's so hard about PDF text extraction?

#243

Post by pqdfdsaff »

emodendroket wrote: post #227
pqdfdsaff wrote: post #182

Earlier quoted context omitted.

If I can produce an image that you incorrectly label as Bird or No Bird, does that mean it's accurate to say you cannot tell me if pictures have birds in them? Or is that needlessly pedantic beyond any practical use case and clearly the intended context?

No, I don't think it's pedantic and impractical. There are many use cases where the possibility of adversarial input makes heuristics a non-starter.

Please tell me the practical concerns regarding adversarial inputs against a phone app that tries to tag pictures of birds in national parks.

Re: What's so hard about PDF text extraction?

#244

Post by hnick »

daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

I work in the print industry and some clients have the naive idea they'll save money by formatting their own documents (naive because usually this just means a lot more work for us, which they end up paying for).

We need some metadata to rearrange and sort PDF pages for mailing and delivery (such as name, address, and start/end page for that customer).

Our general rule is you provide metadata in an external file to make it easy for us. Otherwise, we run pdftotext and hope there's a consistent formatting for the output (e.g. every first page has "Issue Date:", "Dear XYZ,", or something written on it).

If that doesn't work then we're re-negotiating. It is not too difficult usually to build a parser for one family of PDF files based on a common setup as you've said and you get to learn various tricks. It is very difficult though to write a general parser.

Personally, I found parsing postscript easier since usually it was presented linearly.

Re: What's so hard about PDF text extraction?

#245

Post by hnick »

mopsi wrote: post #209
bob1029 wrote: post #69

Earlier quoted context omitted.

We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…

> No one can tell the difference between source and destination PDF unless they look at the file size on disk. Not even when they try to select and copy text?

You can add PDF tag commands to make rasterised text selectable and searchable, though they probably aren't doing that.

Re: What's so hard about PDF text extraction?

#246

Post by hnick »

mikestew wrote: post #51
p0nce wrote: post #26

The take-away is that PDF should not be an input to anything.

Except eyeballs and printers, and printers are just an eyeball abstraction.

You'd hope so, but some printers run some very finicky software with less horsepower than your desktop machine so can fall over on complex PDF structures. I preferred Postscript!

Re: What's so hard about PDF text extraction?

#247

Post by hnick »

alkonaut wrote: post #138
aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

A PDF isn’t for storage it’s for display. It’s the equivalent of a printout. You don’t delete your CAD drawing or spreadsheet after printing it out.

PDF/A is intended for archival and long term storage.

They key difference is it contains no ambiguity (such as all fonts must be embedded).

Re: What's so hard about PDF text extraction?

#248

Post by dandelo1953 »

Alex3917 wrote: post #179
tolmasky wrote: post #169

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Does this mean some abstraction is lost between the creation phase and final "save to pdf" phase? It'd seem ridiculous to not easily be able to track blocks while it's a WIP.....

Re: What's so hard about PDF text extraction?

#249

Post by aasasd »

alkonaut wrote: post #138
aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

A PDF isn’t for storage it’s for display. It’s the equivalent of a printout. You don’t delete your CAD drawing or spreadsheet after printing it out.

> A PDF isn’t for storage it’s for display. It’s the equivalent of a printout.

This conjecture would have some practical relevance if I had access to the same papers in other formats, preferably HTML. Yet I'm saddened time and again to find that I don't.

In fact, producing HTML or PDF from the same source was exactly my proposed route before I was told that apparently Tex is only good for printing or PDFs. I hope that this is false, but not in a position to argue currently.

Re: What's so hard about PDF text extraction?

#250

Post by 72deluxe »

siftrics wrote: post #211
ccozan wrote: post #207

Earlier quoted context omitted.

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

I like how your demo is clear and also in Linux! Surprising.

The page is clear and easy to understand, looks good. Well done.

Post reply on HN