Live data from Hacker News

What's so hard about PDF text extraction?

filingdb.com

191–200 of 350 posts

Re: What's so hard about PDF text extraction?

#191
post #94

Earlier quoted context omitted.

Are you building an app?

Building personal finance app to keep track of multiple bank accounts and investments, categorising spends, etc. Parsing statement PDFs from every bank is pretty hellish.

That’s why the open banking api is amazing these days

In the past we did purposely make it more difficult to parse our PDFs

Re: What's so hard about PDF text extraction?

#192
post #173

Not the only format Adobe made overly complex. Wonderfully amusing code comment, about Adobe PSD format: https://fallenpegasus.livejournal.com/854615.html

Are there 30- to 40-year-old application formats that you think have done a better job adapting to new needs and 4 to 6 orders of magnitude improvements in the systems they run on?

Re: What's so hard about PDF text extraction?

#193

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases.

PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes.

Re: What's so hard about PDF text extraction?

#194

Earlier quoted context omitted.

> Currently, there is no viable alternative if you want the pros but not the cons I remember OpenXPS being much easier to work with. That might be due to cultural rather than structural differences, mind - fewer applications generate OpenXPS, so there's fewer applications to generate them in their own special snowflake ways.

This is the first time I heard of it. When I search for it I only find the Wikipedia article and 99 links to how to convert it to pdf. The problem with this is that from an average person perspective it doesn't have the pros. There is no built-in or first-party app that can open this format on Mac and Linux. More than 99% of the users only want to read or print it. It's hard to convince them to use an alternative for…

It was too late and probably too attached to Microsoft to succeed. It is still used as the spool file format for modern printer drivers on Windows.

Re: What's so hard about PDF text extraction?

#195

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

So I have a lot of experience with basically the same problem just from working on this: https://www.prettyfwd.com. As an example of the opportunity size just in the email domain, the amount of personal non-spam email sent every day is like 100x the total size of Wikipedia, but nothing is really done with any of this information because of this challenge. Basically applications are things like:

- Better search engine results

- Identifying experts within a company

- Better machine translation

- Finding accounting fraud

- Automating legal processes

For context, the reason why Facebook is the most successful social network is that they're able to turn behavioral residue into content. If you can get better at taking garbage data and repackaging it into something useful, it stands to reason that there are lots of other companies the size of Facebook that can be created.

Re: What's so hard about PDF text extraction?

#196
post #58

PDF is, without a doubt, one of the worst file formats ever produced and should really be destroyed with fire... That said, as long as you think of PDF as an image format it's less soul destroying to deal with.

What alternative do you propose? Postscript?

Why not, .ps.gz works pretty well.

Re: What's so hard about PDF text extraction?

#197

This is why iPhone didn't initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted. Edit: To add a little more…

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

“The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, ”

At least :)

Re: What's so hard about PDF text extraction?

#198

I’m a contractor. One of my gigs involved writing parsers for 20-something different kinds of pdf bank statements. It’s a dark art. Once you’ve done it 20 times it becomes a lot easier. Now we simply POST a pdf to my service and it gets parsed and the data it contains gets chucked into a database. You can go extremely far with naive parsers. That is, regex combined with positionally-aware fixed-length formatting rule…

I've written similar code for investment banks, to extract financial reporting data from PDFs. It's shocking to think how much of the financial world runs on this kind of tin-cans-on-a-piece-of-string solution.

Re: What's so hard about PDF text extraction?

#199

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

Almost any business that has physical suppliers or business customers.

PDF is de-facto standard for any invoicing, POs, quotes, etc.

If you solve the problem you can effectively programmatically deal with invoicing/payments/ large parts of ordering/dispensing. It's a no brainer to add it on to almost any financial/procurement software that deals with inter business stuff.

Any small-medium physical business can probably half their financial department if you can dependably solve this issue.

Re: What's so hard about PDF text extraction?

#200

Earlier quoted context omitted.

> I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a "block" (paragraph), and I got a huge explanation on how hard it would be. The funny thing is that creating a universal algorithm to convert PDFs and/or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as…

What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.

The patent office?
Post reply on HN