Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

201–210 of 350 posts

Re: What's so hard about PDF text extraction?

#201

Post by Ididntdothis »

Iwillgetby wrote: post #154
DantesKite wrote: post #143

Earlier quoted context omitted.

Right? I love the OCR for Google Drive. It's such a useful, hidden feature. By the way, why do you wait 10 minutes? Is there a signal that the PDF is done processing? Or is there just some kind of voodoo magic that seems to happen that just takes 10 minutes to do?

2 minutes is probably long enough. I did notice that google drive doesn't seem to like it if you upload a lot of files. I have had files sit and never get OCR, but I forgot about them so they may have OCR on them now. Also, I am not aware of a signal when it is done.

You got to love modern software. It may do it or not. It may do it within an unknowable timeframe. But if it does it, it’s wonderful.

Re: What's so hard about PDF text extraction?

#202

Post by mark-r »

wpietri wrote: post #192
mark-r wrote: post #173

Not the only format Adobe made overly complex. Wonderfully amusing code comment, about Adobe PSD format: https://fallenpegasus.livejournal.com/854615.html

Are there 30- to 40-year-old application formats that you think have done a better job adapting to new needs and 4 to 6 orders of magnitude improvements in the systems they run on?

TIFF did a wonderful job of being forward thinking. It has been the base for a number of other image file formats.

Re: What's so hard about PDF text extraction?

#203

Post by kbenson »

aidos wrote: post #193
Alex3917 wrote: post #179

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases. PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane becaus…

> Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes.

I've actually seen obfuscation used in a PDF where they load in a custom font that changes the character mapping, so the text you get out of the PDF is gibberish, but the fonts displayed on rendering are correct (a simple character substitution cipher).

The important thing to remember whenever you think something should be simple, is that someone somewhere has a business need for it to be more complicated, so you'll likely have to deal with that complication at some point.

Re: What's so hard about PDF text extraction?

#204

Post by zo1 »

shim__ wrote: post #129
LeonM wrote: post #60

Earlier quoted context omitted.

I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down. The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing. Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice. Off course there were workarounds…

I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.

In a way it has. In my experience, there have been multiple times where a "generate PDF" requirement has come up, with the best viable solution being "develop it in HTML using standard tech" followed by "and then convert it to PDF".

Re: What's so hard about PDF text extraction?

#205

Post by bobbylarrybobby »

ProZsolt wrote: post #59
Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

PDF is great what it meant to be, a digital printed paper, with its pros (It will look exactly the same anywhere) and cons (Can't easily extract data from it or modify it). Currently, there is no viable alternative if you want the pros but not the cons

For me, the biggest con of PDFs is that like physical books, the font family and size cannot be changed. This means you can't blow the text up without having to scroll horizontally to read each line or change the font to one you prefer for whatever reason. It boggles my mind that we accept throwing away the raw underlying text that forms a PDF. PDF is one step above a JPEG containing the same contents.

Re: What's so hard about PDF text extraction?

#206

Post by anoncake »

Finnucane wrote: post #56
lm28469 wrote: post #44

Earlier quoted context omitted.

PDF is good at what it's supposed to be good. Parsing pdf to extract data is like using a rock as a hammer and a screw as a nail, if you try hard enough it'll eventually work but it was never intended to be used that way.

Actually, parsing text data from a pdf is more like using the rock to unscrew a screw, in that it was not meant to be done that way at all. But yeah, the pdf was designed to provide a fixed-format document that could be displayed or printed with the same output regardless of the device used. I'm not sure (I haven't thought about it a lot) that you could come up with a format that duplicates that function and is also…

It's closer to using a screwdriver to screw in a rock. The task isn't supposed to be done in the first place but the tool is the least wrong one.

Re: What's so hard about PDF text extraction?

#207

Post by ccozan »

kbenson wrote: post #203
aidos wrote: post #193

Earlier quoted context omitted.

I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases. PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane becaus…

> Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes. I've actually seen obfuscation used in a PDF where they load in a custom font that changes the character mapping, so the text you get out of the PDF is gibberish, but the fonts displayed on rendering are correct (a simple character substitu…

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

Re: What's so hard about PDF text extraction?

#208

Post by Fnoord »

shim__ wrote: post #129
LeonM wrote: post #60

Earlier quoted context omitted.

I build such a service, but it is impossible to guarantee any reliable result. I ended up shutting it down. The PDF standard is a mess, and the number of 'tricks' I've seen done is astonishing. Example: to add shade or border effect to text, most PDF generators simple add the text twice with a subtle offset and different colors. Result: your SaaS service returns every sentence twice. Off course there were workarounds…

I'm actually surprised that PDF hasn't been superseded by some form of embedded HTML by now.

It partly has: ePub [1], an open format for ebooks, contain HTML.

[1] https://en.wikipedia.org/wiki/EPUB

Re: What's so hard about PDF text extraction?

#209

Post by mopsi »

bob1029 wrote: post #69
Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

We have to fill existing PDFs from a wide range of vendors and clients. Our approach is to raster all PDFs to 300DPI PNG images before doing anything with them. Once you have something as a PNG (or any other format you can get into a Bitmap), throwing it against something like System.Drawing in .NET(core) is trivial. Once you are in this domain, you can do literally anything you want with that PDF. Barcodes, images,…

> No one can tell the difference between source and destination PDF unless they look at the file size on disk.

Not even when they try to select and copy text?

Re: What's so hard about PDF text extraction?

#210

Post by aidos »

kbenson wrote: post #203
aidos wrote: post #193

Earlier quoted context omitted.

I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases. PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane becaus…

> Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes. I've actually seen obfuscation used in a PDF where they load in a custom font that changes the character mapping, so the text you get out of the PDF is gibberish, but the fonts displayed on rendering are correct (a simple character substitu…

That’s actually often just a consequence of the subsetting (I think). Believe it or not, you can often rebuild the cmaps using information in the pdf to fix the mapping and make the extraction work again.
Post reply on HN