Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

141–150 of 350 posts

Re: What's so hard about PDF text extraction?

#141

Post by dunham »

giovannibonetti wrote: post #97

One of the main features of the product I work on is data extraction from a specific type of PDF. If you want to build something similar these are my recommendations for you: - Use https://github.com/flexpaper/pdf2json to convert the PDF in an array of (x, y, text) tuples - Use a good text parsing library. Regexes are probably not enough for your use case. In case you are not aware of the limitations of regexes you m…

For simple cases, I've also found "pdftotext -layout" useful. For a quick on-off job, this would save someone the trouble of assembling the lines themselves.

I have used this in the past to extract tables, but it doesn't help much in cases where you need font size information.

Re: What's so hard about PDF text extraction?

#142

Post by dredmorbius »

bsdubernerd wrote: post #87

It's nice to note how several of these problems already exist in much more structured document types, such as HTML. Using white-on-white dark-hat SEO techniques for keyword boosting? Check. Custom fonts with random glyphs? Check. I didn't see custom encodings (yet). We try to keep HTML semantic, but google has been interpreting pages to a much higher level in order to spot issues such as these. If you ever tried to w…

Hiding page content unless rendered via JS is the darkest dark pattern in HTML I've noted.

Though absolute-positioning of all text elements via CSS at some arbitrary level (I've seen it by paragraph), such that source order has no relationship to display order, is quite close.

Re: What's so hard about PDF text extraction?

#143

Post by DantesKite »

Iwillgetby wrote: post #80

If you upload a pdf to google drive and download it 10 minutes later it will magically have BY FAR the best OCR results in the pdf. Note my pdf tests were fairly clean so your experience may not be the same. I have used Google's fine OCR results to simulate a hacker. - Download a youtube video that shows how to attack a server on the website hackthebox.eu - Run ffmpeg to convert the video to images. - Run a jpeg to p…

Right? I love the OCR for Google Drive. It's such a useful, hidden feature.

By the way, why do you wait 10 minutes? Is there a signal that the PDF is done processing?

Or is there just some kind of voodoo magic that seems to happen that just takes 10 minutes to do?

Re: What's so hard about PDF text extraction?

#144

Post by bruckie »

jazzido wrote: post #134
staticautomatic wrote: post #124

Earlier quoted context omitted.

Edit: See reply below Am I reading the repos correctly? It looks like Extractable copied Tabula (MIT) to its own repo rather than forking it, removed the attribution, and then tried to re-license it as Apache 2.0. If so, that would be pretty fucked up. https://github.com/tabulapdf https://github.com/ExtractTable/tabulapro

Not really. They import tabula_py, which is a Python wrapper around tabula-java (the library of which I'm a maintainer). Still, I would have loved at least a heads up from the team that sells Tabula Pro. I know they're not required to do so, but hey, they're kinda piggybacking on Tabula's "reputation".

If you control the Tabula trademark (which doesn't necessarily require a formal registration), you may be able to prohibit them from using the TabulaPro name. That's exactly what trademark law is for.

(IANAL)

Re: What's so hard about PDF text extraction?

#145

Post by LorenPechtel »

p0nce wrote: post #26

The take-away is that PDF should not be an input to anything.

Unfortunately, when you need the output of program A as the input to B sometimes you have to jump through such hoops. I've never done it with .pdf but I've fought similar battles with .xps and never fully conquered them. (And the parser was unstable as hell, besides--it would break with every version and sometimes for far lesser reasons.)

Re: What's so hard about PDF text extraction?

#146

Post by LorenPechtel »

Santosh83 wrote: post #2

Another site that breaks the browser's back navigation. Why do so many sites do this? Do they imagine they retain user attention for longer if they break navigation? It's pretty trivial to long-press the back button or just close the tab and not come back again to your site...

Broken back buttons are often due to a site where a placeholder loads and it turns around and loads the real thing. Back takes you to the placeholder which promptly takes you back where you were.

There are sites that explicitly mess with the back button but I haven't seen one in ages.

Re: What's so hard about PDF text extraction?

#147

Post by totololo »

eterps wrote: post #122
Wiretrip wrote: post #38

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

Lots of people doing their daily jobs are not aware of the information loss that occurs whenever they are saving/exporting as PDF.

In the consulting industry I’ve seen PDF being used precisely because third parties couldn’t mess with the content anymore.

Re: What's so hard about PDF text extraction?

#149

Post by dredmorbius »

the_french wrote: post #47

Is there a tool that works for the limited subset of PDFs generated by Latex? Do those documents have more structure than the average PDF? Less? It'd be nice to extract text from scientific articles at least.

I've had remarkably good results in general (for reading) using the Poppler library's "pdftotext" utility. Since it defaults to writing output to file, I wrap that in a bash function to arrive at a less-like pager, with page breaks noted:

    pdfless ()
    {
       pdftotext -layout "$1" - |
       sed 's/\f/\n\n ----------------- -----------------  ----------------- ----------------- \n\n\n/g' |
       ${PAGER:-less -S}
    }
The key is the "-layout" argument, which preserves original layout of the document. This ... may not be what you want visually, but makes backing out the original text somewhat easier.

Of course, requesting the LaTeX sources would be preferred.

Post reply on HN