Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

281–290 of 350 posts

Re: What's so hard about PDF text extraction?

#281

Post by siftrics »

malikolivier wrote: post #276
siftrics wrote: post #211

Earlier quoted context omitted.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

How is support for languages other than English? I am especially thinking about Japanese. Our company could probably find good uses of such service if it had Japanese support.

Yes, Japanese is supported! Almost all languages are supported.

If you have any questions or need help trying it out, please email me at siftrics@siftrics.com. We can hop on the phone, too, if you'd like.

Re: What's so hard about PDF text extraction?

#282

Post by BashiBazouk »

It's interesting to see other views of PDF. As someone who lives in Illustrator ripping every little piece of data out of a pdf to import into an Illustrator or InDesign file and then making a production pdf for large format printing and fixing plenty of issues along the way I find the text almost inconsequential to the whole thing. It's just another element among many elements: images, vector illustrations, et. PDF might not the best way to pass along pure text but as a container for graphical representation it works pretty well. I build pdf files to describe 20 ft walls with 1+ gigabyte images, complex vector illustrations, finely formatted text and it all prints out dam close to how I planned for it down to exact colors that match specific Pantone swatches. It's amazing what can be packed in to a pdf...

Re: What's so hard about PDF text extraction?

#283

Post by bjonnh »

tolmasky wrote: post #187
Alex3917 wrote: post #179

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Academics and other people that rely on scientific publications. Most of the world's knowledge in science is locked into PDFs and screenshots (or even pictures) of manufacturer's (often proprietary) software... So extracting it in a more structured way would be a win (so HTML may not be best). On a related note, I've seen people using Okular to convert PDF tables to a usable form (to be honest its table extraction tool is one of the best i've seen despite being pretty manual).

Re: What's so hard about PDF text extraction?

#284

Post by aasasd »

dang wrote: post #273
aasasd wrote: post #161

Earlier quoted context omitted.

So you're saying that if you have HTML as a tool then you're irresistibly drawn to making a complex page with menus and ugliness? I don't think that's a problem in HTML. I think the problem is in your head. > we need a format that stays the same size, same proportions Please explain how this helps people read a paper on screens of different sizes and proportions. Like me on my phone.

Please don't cross into personal attack in HN comments. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html

It's pretty obviously not a personal attack.

Re: What's so hard about PDF text extraction?

#285

Post by wpietri »

saradhi wrote: post #279
wpietri wrote: post #189

Earlier quoted context omitted.

You're being much more polite here than I would be. Even if it isn't illegal, what they've done is a giant dick move.

William, the intention of "TabulaPro" is to give the developers a chance to use a single library instead of switching ExtractTable for images and tabula-py for text PDFs. What do you recommend us to do, to not make you feel we made a dick move. TIA

Well, let me ask a few questions:

Did you ask permission of the original author to use a derived name?

Did you discuss your plan to commercialize the original author's work with the author? Before starting out?

Since starting a commercial project, how much money have you given to the original author?

Re: What's so hard about PDF text extraction?

#286

Post by wpietri »

mark-r wrote: post #202
wpietri wrote: post #192

Earlier quoted context omitted.

Are there 30- to 40-year-old application formats that you think have done a better job adapting to new needs and 4 to 6 orders of magnitude improvements in the systems they run on?

TIFF did a wonderful job of being forward thinking. It has been the base for a number of other image file formats.

I think TIFF has a number of advantages there. It was from the beginning an interchange format, so it had the opportunity to look at a bunch of existing formats and extract the commonality. It's also not an application format, so the pace of change is slower and more controlled; it can trail rather than lead. And it is of course a standard, which means a different set of dynamics around how things get added and how clear the specs have to be.

That's not to say it isn't great; I could well believe it. But I'm just not shocked that PSD and PostScript have ended up being a bit of a mess over the decades. I doubt I could have done any better.

Re: What's so hard about PDF text extraction?

#287

Post by aasasd »

lmm wrote: post #269
aasasd wrote: post #89

What a glorious format for storing mankind's knowledge. Consider that by now displays have arbitrary sizes and a variety of proportions, and that papers are often never printed but only read from screens. To reflow text for different screen sizes, you need its ‘semantic’ structure. And meanwhile if you say on HN that HTML should be used instead of PDF for papers, people will jump on you insisting that they need PDF f…

HTML has all the same problems and degrades over time. A PDF from 20 years ago will at least be readable by a human; a HTML page does not even guarantee that much. You're right that most of the relevant semantics would fit into Markdown. So store the markdown! There are problems with PDF but HTML is the worst of all worlds.

What exactly degrades about HTML in twenty years? I can read pages from the 90s just fine: the main thing off is the font size due to the change in screen resolutions, but—surprise!—plain HTML scales and reflows beautifully on big and small screens. (Which is the complete opposite of ‘HTML has the same problems’.) I hope you're not lamenting the loss of the ‘blink’ tag.

If you're talking about images and whatnot falling off, that's a problem of delivery and not the format.

Markdown translates to HTML one-to-one, it's in the basic features of Markdown. For some reason I have to repeat time and again: use a subset of HTML for papers, not ‘glamor magazine’ formatting. The use of HTML doesn't oblige you to go wild with its features.

Re: What's so hard about PDF text extraction?

#288

Post by aasasd »

72deluxe wrote: post #256
aasasd wrote: post #180

Earlier quoted context omitted.

I wonder if you realize that both your points wildly miss what I said. First, there's no need to stretch my argument to the point of it being ridiculous. I don't have to reach for a watch to suffer from PDF. Even a tablet is enough: I don't see many 14" tablets flying off the shelves. I also know for sure that the vast majority of ubiquitous communicator devices , aka smartphones, are about the same size as mine, so…

Some people are very "funny" about the layout of items and text and want it to be preserved identically to their "vision" when they created it. For example, every "marketing" individual when they see a webpage seem to want it pixel-perfect. I think it's the artist in them. This is understandable in some instances: a. Picasso's or Monet's works probably wouldn't be as good if you just roll them up into a ball. Sure, t…

That's a tradeoff between complex formatting and accessibility of the result. Authors are making readers sit in front of desktops/laptops for some wins in formatting. Considering that papers, at least ones that I see, are all just columns of text, images and formulas, the win seems to be marginal, while the loss in accessibility is maddening with the current tech-ecosphere.

Re: What's so hard about PDF text extraction?

#289

Post by aasasd »

dang wrote: post #273
aasasd wrote: post #161

Earlier quoted context omitted.

So you're saying that if you have HTML as a tool then you're irresistibly drawn to making a complex page with menus and ugliness? I don't think that's a problem in HTML. I think the problem is in your head. > we need a format that stays the same size, same proportions Please explain how this helps people read a paper on screens of different sizes and proportions. Like me on my phone.

Please don't cross into personal attack in HN comments. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html

By the way, the ‘before’ one also doesn't classify as a personal attack for me. Not sure what your definition of a personal attack is.

Re: What's so hard about PDF text extraction?

#290

Post by tastyminerals »

greycol wrote: post #199
tolmasky wrote: post #187

Earlier quoted context omitted.

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Almost any business that has physical suppliers or business customers. PDF is de-facto standard for any invoicing, POs, quotes, etc. If you solve the problem you can effectively programmatically deal with invoicing/payments/ large parts of ordering/dispensing. It's a no brainer to add it on to almost any financial/procurement software that deals with inter business stuff. Any small-medium physical business can probab…

Yes, most invoices are in PDF but only about 40% of them are native PDF meaning they are actual documents not scanned images converted to PDFs. There are are also compound PDF invoices which contain images. So, in order to extract data from them, one needs not only good PDF parser but an OCR engine too.
Post reply on HN