Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

Post reply on HN
Print view View original

341–350 of 350 posts

Re: What's so hard about PDF text extraction?

#341

Post by aasasd »

lmm wrote: post #339
aasasd wrote: post #287

Earlier quoted context omitted.

What exactly degrades about HTML in twenty years? I can read pages from the 90s just fine: the main thing off is the font size due to the change in screen resolutions, but—surprise!—plain HTML scales and reflows beautifully on big and small screens. (Which is the complete opposite of ‘HTML has the same problems’.) I hope you're not lamenting the loss of the ‘blink’ tag. If you're talking about images and whatnot fall…

> What exactly degrades about HTML in twenty years? I can read pages from the 90s just fine: the main thing off is the font size due to the change in screen resolutions, but—surprise!—plain HTML scales and reflows beautifully on big and small screens. (Which is the complete opposite of ‘HTML has the same problems’.) I hope you're not lamenting the loss of the ‘blink’ tag. I am indeed, and of other tags that are no lo…

> Unless you can clearly define the line between which features should be used and which should not, you'll end up with all of the features of HTML being used, and all of the problems that result.

You'll notice that I said in the top-level comment that my beef is with PDF papers (i.e. scientific and tech). I don't care about magazines and such, since they obviously have different requirements. So let's transfer your argument to current papers publishing:

“Since PDF can format text and graphics in arbitrary ways, you'll end up with papers that look like glamor and design magazines and laid out like Principia Discordia and Dada posters. You'll have embedded audio, video and 3D objects since PDF supports those, and since it can embed Flash apps, you'll have e.g. ‘RSS Reader, calculator, and online maps’ as suggested by Adobe, and probably also games. PDF also has Javascript and interactive input forms, so papers will be dynamic and interactive and function as clients to web servers.”

You can decide for yourself whether this corresponds to reality, and if the hijinks of CSS2-era websites are relevant.

What is it with people, one after another, jumping to the same argument of ‘if authors have HTML, they will immediately go bonkers’? If really looks like some Freudian transfer of innate tendencies. We have Epub, for chrissake, which is zipped HTML—what, have Epub books gone full Dada while I wasn't looking? Most trouble with Epub that I've had is inconvenience with preformatted code.

> Old sites are often impossible to render with the correct layout. Resources refuse to load because of mixed-content policy or because they're simply gone - which is a problem with the format because the format is not built for providing the whole page as a single artifact.

Yes, as I mentioned under the link provided in the top-level comment, the non-use of a packaged-HTML delivery is precisely my beef here. The entire idea of using HTML for papers implies employing a package format, since papers are usually stored locally. It's a chicken-and-egg problem. It's solved by the industry picking one of the dozen available package formats and some version of HTML for the content. Which would still mean that HTML is used for formatting. HTML could be embedded in PDF for all I care, if I can sanely read the damn thing on my phone.

Re: What's so hard about PDF text extraction?

#342

Post by efreak »

goatlover wrote: post #240
BlueTemplar wrote: post #223

Earlier quoted context omitted.

Except it sucks for reading?

I haven't experienced problems reading articles and books in PDF format on my phone.

I read ebooks on my Nintendo DSi for several years when I was in college; The low-resolution screen combined with my need for glasses (and dislike of wearing them) made reading PDF files unbearable. Later on I got a cheap android tablet and reading PDF files was easier, but still required constant panning and zooming. Today I use a more modern device (2013 Nexus 7 or 2014 NVidia Shield), and I still don't like PDF files. I usually open the PDF in word if possible, save it in another format, then convert to epub with calibre, and dump the other formats.

Epubs in comparison are easy, as all it takes is a single tap or button press to continue. When there's no DRM on the file (thanks HB, Baen) I read in FBReader with custom fonts, colors, and text size. It doesn't hurt any that the epub files I get are usually smaller than the PDF version of the same book.

Personally, I think the fact that Calibre's format converter has so many device templates for PDF conversion says a lot.

Re: What's so hard about PDF text extraction?

#343

Post by lmm »

aasasd wrote: post #341
lmm wrote: post #339

Earlier quoted context omitted.

> What exactly degrades about HTML in twenty years? I can read pages from the 90s just fine: the main thing off is the font size due to the change in screen resolutions, but—surprise!—plain HTML scales and reflows beautifully on big and small screens. (Which is the complete opposite of ‘HTML has the same problems’.) I hope you're not lamenting the loss of the ‘blink’ tag. I am indeed, and of other tags that are no lo…

> Unless you can clearly define the line between which features should be used and which should not, you'll end up with all of the features of HTML being used, and all of the problems that result. You'll notice that I said in the top-level comment that my beef is with PDF papers (i.e. scientific and tech). I don't care about magazines and such, since they obviously have different requirements. So let's transfer your…

> “Since PDF can format text and graphics in arbitrary ways, you'll end up with papers that look like glamor and design magazines and laid out like Principia Discordia and Dada posters. You'll have embedded audio, video and 3D objects since PDF supports those, and since it can embed Flash apps, you'll have e.g. ‘RSS Reader, calculator, and online maps’ as suggested by Adobe, and probably also games. PDF also has Javascript and interactive input forms, so papers will be dynamic and interactive and function as clients to web servers.”

Those things don't happen in PDFs in the wild, or at least not to any great extent. It's not that technical paper authors have shown some special restraint and limited themselves to a subset of what the rest of the PDF world does. Technical papers look much like any other PDF and do much the same thing that any other PDF does; if they were authored in HTML, we should expect them to look much like any other HTML page and do much the same thing that any other HTML page does. Based on my experience of HTML pages, that would be a massive regression.

> Yes, as I mentioned under the link provided in the top-level comment, the non-use of a packaged-HTML delivery is precisely my beef here. The entire idea of using HTML for papers implies employing a package format, since papers are usually stored locally. It's a chicken-and-egg problem. It's solved by the industry picking one of the dozen available package formats and some version of HTML for the content. Which would still mean that HTML is used for formatting. HTML could be embedded in PDF for all I care, if I can sanely read the damn thing on my phone.

The details matter; you can't just handwave the idea of a sensible set of restrictions and a good packaging format, because it turns out those concepts mean very different things to different people. If you want to talk about, say, Epub, then we can potentially have a productive conversation about how practical it is to format papers adequately in the Epub subset of CSS and how useful Epub reflowing is versus how often a document doesn't render correctly in a given Epub reader. If all you can say about your proposal is "a subset of HTML" then of course people will assume you're proposing to use the same kind of HTML found on the web, because that's the most common example of what "a subset of HTML" looks like.

Re: What's so hard about PDF text extraction?

#344

Post by aasasd »

lmm wrote: post #343
aasasd wrote: post #341

Earlier quoted context omitted.

> Unless you can clearly define the line between which features should be used and which should not, you'll end up with all of the features of HTML being used, and all of the problems that result. You'll notice that I said in the top-level comment that my beef is with PDF papers (i.e. scientific and tech). I don't care about magazines and such, since they obviously have different requirements. So let's transfer your…

> “Since PDF can format text and graphics in arbitrary ways, you'll end up with papers that look like glamor and design magazines and laid out like Principia Discordia and Dada posters. You'll have embedded audio, video and 3D objects since PDF supports those, and since it can embed Flash apps, you'll have e.g. ‘RSS Reader, calculator, and online maps’ as suggested by Adobe, and probably also games. PDF also has Java…

> It's not that technical paper authors have shown some special restraint and limited themselves to a subset of what the rest of the PDF world does. Technical papers look much like any other PDF and do much the same thing that any other PDF does.

This makes zero sense to me. You're saying that technical papers look the same as Principia Discordia or glamor/design magazines or advertising booklets, including those that just archive printed media. That technical papers include web-form functionality just like some PDFs do—advertising or whatnot, I'm not sure. If that's the reality for you then truly I would abhor living in it—guess I'm relatively lucky here in my world.

However, if you point me to where such papers hang out, I would at least finally learn what mysterious ‘complex formatting’ people want in papers and which can only be satisfied by PDF.

Re: What's so hard about PDF text extraction?

#345

Post by duckmysick »

jiveturkey wrote: post #121
sixhobbits wrote: post #9

As a meta point, it's really nice to see such a well-written, well-researched article that is obviously used as a form of lead generation for the company, and yet no in your face "call to actions" which try to stop you reading the article you came for and get out your wallet instead.

i mean except for the banner at the top and bottom! but yeah, an SEO article with actual substance, well formatted, not grey-on-grey[1], no trackers [2], is rare these days. [1] recently read an SEO post on okta's site. who can read that garbage? [2] only GA ... which isn't a 3rd-party tracker.

> GA ... which isn't a 3rd-party tracker.

Why not? It's not self-hosted and results are stored elsewhere.

Re: What's so hard about PDF text extraction?

#346

Post by daniel-levin »

graeme wrote: post #160
daniel-levin wrote: post #25

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

What’s your contact info? Didn’t see any on your github.

dan at threeaddresscode dot com

Re: What's so hard about PDF text extraction?

#347

Post by emodendroket »

pqdfdsaff wrote: post #243
emodendroket wrote: post #227

Earlier quoted context omitted.

No, I don't think it's pedantic and impractical. There are many use cases where the possibility of adversarial input makes heuristics a non-starter.

Please tell me the practical concerns regarding adversarial inputs against a phone app that tries to tag pictures of birds in national parks.

Well of course if you define it narrowly that's true, but computer vision has many more safety-critical applications, most obviously self-driving.

Re: What's so hard about PDF text extraction?

#348

Post by jiveturkey »

duckmysick wrote: post #345
jiveturkey wrote: post #121

Earlier quoted context omitted.

i mean except for the banner at the top and bottom! but yeah, an SEO article with actual substance, well formatted, not grey-on-grey[1], no trackers [2], is rare these days. [1] recently read an SEO post on okta's site. who can read that garbage? [2] only GA ... which isn't a 3rd-party tracker.

> GA ... which isn't a 3rd-party tracker. Why not? It's not self-hosted and results are stored elsewhere.

it doesn't correlate across sites by default -- the reasonable definition of a 3rd party tracker. by your definition, everything not complete self-hosted is a 3rd-party tracker. eg, netlify, which uses server logs to "self"-analyze would be a 3rd party tracker. it is not self-hosted and the data is stored elsewhere.

some might add: for the purpose of resale of the data, but I don't think that's a requirement to be classified as 3rd party tracker. the mere act of correlation, no matter what you then do with the data, makes you a 3rd party tracker. in case you think that's just semantics, this is important for GDPR and the new california law.

you can turn on the "doubleclick" option, which does do said correlation and tracks you. but that's up to the site to decide. GA doesn't do it by default.

Re: What's so hard about PDF text extraction?

#349

Post by lwhsiao »

bhl wrote: post #218
lwhsiao wrote: post #190

Earlier quoted context omitted.

One of the co-authors of Fonduer here. Just for reference the original paper for Fonduer is here: https://dl.acm.org/doi/pdf/10.1145/3183713.3183729 And additional follow-up work on extracting data from PDF datasheets is here: https://dl.acm.org/doi/pdf/10.1145/3316482.3326344 One thing to point out about our library is that while we do take PDF as input and use it to calculate visual features, we also rely on an HTM…

What type of visual features are you looking at? I've been trying to find a web-clipper that uses both visual and structural cues from the rendered page and HTML, but have no luck finding a good starting point.

There are a handful. We looks at bounding boxes to featurize which spans are visually aligned with other spans. Which page a span is on, etc. You can see more in the code at [1]. In general, visual features seem to give some nice redundancy to some of the structural features of HTML, which helps when dealing with an input as noisy as PDF.

[1]: https://github.com/HazyResearch/fonduer/tree/master/src/fond...

Re: What's so hard about PDF text extraction?

#350

Post by whatisthetruth »

I saw this and thought it might be something my company could use so I contacted them for info.

The CEO Simon Mahony basically told me to piss off when I told him I thought the site was misleading since there was no product or service to directly purchase. They make custom developed software that you must pay their consultants to integrate. I would not do business with such a company that acts so unprofessionally even if they have a decent team.

Post reply on HN