Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

211–220 of 350 posts

Re: What's so hard about PDF text extraction?

#211
post #207

Earlier quoted context omitted.

> Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes. I've actually seen obfuscation used in a PDF where they load in a custom font that changes the character mapping, so the text you get out of the PDF is gibberish, but the fonts displayed on rendering are correct (a simple character substitu…

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use.

If you care to check us out: https://siftrics.com/

Re: What's so hard about PDF text extraction?

#212

Earlier quoted context omitted.

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

So I have a lot of experience with basically the same problem just from working on this: https://www.prettyfwd.com . As an example of the opportunity size just in the email domain, the amount of personal non-spam email sent every day is like 100x the total size of Wikipedia, but nothing is really done with any of this information because of this challenge. Basically applications are things like: - Better search engin…

I often ponder how much of the "old world" will get "digitalized" — translated in numeric form, bits. And how much will just disappear. The question might seem trivial if you think of books, but now think of architecture, language itself (as it evolves), etc.

There's almost no question in my mind that most new data will endure in some form, by virtue of being digital from day 1.

The endgame for such a company, imho, is to become the "source entity" of information management (in abstracted form), whose two major products are one to express this information in the digital space, and the other in the analog/physical space. You may imagine variations of both (e.g. AR/VR for the former).

Kinda like language in the brain is "abstract" (A) (concept = pattern of neurons firing) and then speech "translates" into a given language, like English (B) or French (C) (different sets of neurons). So from A you easily go to either B or C or D... We've observed that Deep Learning actually does that for translation (there's a "new" "hidden" language in the neural net that expresses all human languages in a generic form of sorts, i.e. "A" in the above example).

The similarities of the ontology of language, and the ontology of information in a system (e.g. business) are remarkable — and what you want is really this fundamental object A, this abstract form which then generates all possible expressions of it (among which a little subset of ~1,000 gives you human languages, a mere 300 active iirc; and you might extend that into any formal language fitting the domain, like engineering math/physics, programming code, measurements/KPI, etc.

It's a daunting task for sure but doable because the space is highly finite (nothing like behavior for instance; and you make it finite through formalization, provided your first goal is to translate e.g. business knowledge, not Shakespeare). It's also a one-off thing because then you may just iterate (refine) or fork, if the basis is sound enough.

I know it all sounds sci-fi but having looked at the problem from many angles, I've seen the PoC for every step (notably linguistics software before neural nets was really interesting, producing topological graphs in n dimensions of concepts e.g. by association). I'm pretty sure that's the future paradigm of "information encoding" and subsequent decoding, expression.

It's just really big, like telling people in the 1950's that because of this IBM thing, eventually everybody will have to get up to speed like it's 1990 already. But some people "knew", as in seeing the "possible" and even "likely". These were the ones who went on to make those techs and products.

Re: What's so hard about PDF text extraction?

#213

Earlier quoted context omitted.

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Almost any business that has physical suppliers or business customers. PDF is de-facto standard for any invoicing, POs, quotes, etc. If you solve the problem you can effectively programmatically deal with invoicing/payments/ large parts of ordering/dispensing. It's a no brainer to add it on to almost any financial/procurement software that deals with inter business stuff. Any small-medium physical business can probab…

A business that invests in building a machine that reads data, produced by a 3rd party machine, using format intended for lay humans to read, is not investing in the right tech IMO.

Small-mediums should be looking to consolidate buying through a few good suppliers and working with them directly to automate process, or adopting interchange formats.

Problem for some small-business is the cost (process changes, licencing etc) of adopting interchange formats and working with large vendors is prohibitive at their scale e.g. the airline BSP system.

I agree that solving the problem generally i.e. replacing an accounts payable staff person capable of processing arbitrary invoice documents will be comparable to self-driving in difficulty.

If a company deals with a lot of a single type of PDF, then the approach could be economical. I am actually involved in a project looking at doing this with AWS Textract.

Re: What's so hard about PDF text extraction?

#214
post #207

Earlier quoted context omitted.

That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

Will check it out thanks, we had evaluated ABBYY but it didn't suit. Does your product do something like key value or table detection?

Re: What's so hard about PDF text extraction?

#215

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Organisations that have existing business processes to publish to print and pdf but now want to publish in responsive formata for mobile or even desktop web.

Changing their process might be more expensive than paying a lot of money for them to carry on as is for a few more years while getting the benefit of modern eyes on their content.

Edit: concrete example would be government publications like budget narrative documents.

Re: What's so hard about PDF text extraction?

#216

Earlier quoted context omitted.

I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use. If you care to check us out: https://siftrics.com/

Will check it out thanks, we had evaluated ABBYY but it didn't suit. Does your product do something like key value or table detection?

We do table recognition and pride ourselves on being better at it than ABBYY. We can handle variable number of rows in a table and we take that into account when determining the position of other text on the page.

Feel free to email me at siftrics@siftrics.com with any questions. We can setup a phone call, zoom meeting, or google hangouts too, if you’d like.

Re: What's so hard about PDF text extraction?

#217
For academic papers: GROBID [0] is a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured XML/TEI encoded documents with a particular focus on technical and scientific publications.

[0] https://github.com/kermitt2/grobid/

Re: What's so hard about PDF text extraction?

#218
post #136

There is a fairly interesting library developed by the Stanford Team behind https://www.snorkel.org/ that takes structured documents, including PDF formatted as tables, and builds a knowledge base: https://github.com/HazyResearch/fonduer It looks promising for these kinds of daunting tasks

One of the co-authors of Fonduer here. Just for reference the original paper for Fonduer is here: https://dl.acm.org/doi/pdf/10.1145/3183713.3183729 And additional follow-up work on extracting data from PDF datasheets is here: https://dl.acm.org/doi/pdf/10.1145/3316482.3326344 One thing to point out about our library is that while we do take PDF as input and use it to calculate visual features, we also rely on an HTM…

What type of visual features are you looking at? I've been trying to find a web-clipper that uses both visual and structural cues from the rendered page and HTML, but have no luck finding a good starting point.

Re: What's so hard about PDF text extraction?

#219

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

Semantic segmentation to identify blocks and OCR to convert to text - I think OneNote is already doing that. PDF is a horrible format for representing text, though PostScript is even worse.
Post reply on HN