coincidentally, posted this over on Show HN today. OCR workbench, AI OCR & editing tools for OCRing old / hard documents. https://news.ycombinator.com/item?id=43976450. Tesseract works fine for modern text documents, but it fails badly on older docs (e.g. colonial american, etc)
PDF to Text, a challenging problem
121–130 of 206 posts
Re: PDF to Text, a challenging problem
#122One thing I wish someone would write is something like the browser's developer tools ("inspect elements") for PDF — it would be great to be able to "view source" a PDF's content streams (the BT … ET operators that enclose text, each Tj operator for setting down text in the currently chosen font, etc), to see how every “pixel” of the PDF is being specified/generated. I know this goes against the current trend / state-…
I assume you mean open source or free, but just noting Acrobat Pro was almost there when I last used it years ago. The problem was you had it in reverse, browsing the content tree not inspecting the page, but it did highlight the object on the page. Not down to the command though, just the object/stream.
Re: PDF to Text, a challenging problem
#123PDF is a display format. It is optimised for eyeballs and printers. There has been some feature creep. It is a rubbish machine data transfer mechanism but really good for humans and say storing a page of A4 (letter for the US).
So, you start off with the premise that a .pdf stores text and you want that text. Well that's nice: grow some eyes!
Otherwise, you are going to have to get to grips with some really complicated stuff. For starters, is the text ... text or is it an image? Your eyes don't care and will just work (especially when you pop your specs back on) but your parser is probably seg faulting madly. It just gets worse.
PDF is for humans to read. Emulate a human to read a PDF.
Re: PDF to Text, a challenging problem
#124As someone who has worked on this FT. (S&P, parsing of financial disclosures)
The solution is OCR. Don't fuck with internal file format. PDF is designed to print/display stuff, not to be parseable by machines.
Re: PDF to Text, a challenging problem
#125Why hasn't the PDF standard been replaced or revised to require the text in meta form? Seems like a no brainer.
Re: PDF to Text, a challenging problem
#126Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…
It's still fresh with me, 7 or 8 years ago in my 20s, perhaps you are a bit older? Otherwise wouldn't hurt to do a checkup with a physician
Re: PDF to Text, a challenging problem
#127Below is a PDF. It is a .txt file. I can save it with a .pdf extension and open it in a PDF viewer. I can make changes in a text editor. For example, by editing this text file, I can change the text displayed on the screen when the PDF is opened, the font, font size, line spacing, the maximum characters per line, number of lines per page, the paper width and height, as well as portrait versus landscape mode. %PDF-1.4…
It can also have embedded binary streams. It was not made for text. It was made for layout and graphics. You give nice examples, but each of those lines could have been broken up into one call per character, or per word, even out of order.
It can also use fonts which map glyphs via characters which do not represent the final visual item e.g. "PDF" could be "1#F" and you only really know what it looks like by rendering then viewing/OCR.
A nice file won't, but sometimes the best work is in not dealing with nice things.
Re: PDF to Text, a challenging problem
#128I built a simple OSS tool for qualitative data analysis, which needs to turn uploaded documents into text (stripped HTML). PDFs have been a huge problem from day one.
I have investigated many tools, but two-column layouts and footers etc often still mess up the content.
It's hard to convince my (often non-technical) users that this is a difficult problem.
Re: PDF to Text, a challenging problem
#129The better solution is to embed, in the PDF, the editable source document. This is easily done by LibreOffice. Embedding it takes very little space in general (because it compresses well), and then you have MUCH better information on what the text is and its meaning. It works just fine with existing PDF readers.
That's true, but it's dependent on the creator of the PDF having aligned incentives with the consumer of the PDF. In the e-Discovery field, it's commonplace for those providing evidence to dump it into a PDF purely so that it's harder for the opposing side's lawyers to consume. If both sides have lots of money this isn't a barrier, but for example public defenders don't have funds to hire someone (me!) to process the…
> The solution is to make it illegal to do this: wiretap data, for example, should be provided in a standardized machine-readable format. There's no ethical reason for simple technical friction to be affecting the outcomes of criminal proceedings.
I can’t speak to wiretaps specifically, but when it comes to the legal field, this is usually already how it operates. GDPR, for example, makes specific provisions that user data must be provided in an accessible, machine-readable format. Most jurisdictions also aren’t going to look kindly on physical document dumping and will require that documents be provided in a machine-readable format. PDF is the legal industry standard for all outbound files. The consistency of its formatting makes up for the difficulties involved with machine-readability.
There’s not a huge incentive to find an alternative because most firms will just charge a markup on the time a clerk spends reading through and transcribing those PDFs. If cost is a concern, though, most jurisdictions will require the party in possession of the original documents to provide them in a machine-readable format (e.g. providing bank records as Excel spreadsheets rather than as PDFs).
Re: PDF to Text, a challenging problem
#130Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…
It's still fresh with me, 7 or 8 years ago in my 20s, perhaps you are a bit older? Otherwise wouldn't hurt to do a checkup with a physician
No I'm just good at forgetting things, I've been practicing since I was a kid.