Reminds me of github.com/docwire/docwire
PDF to Text, a challenging problem
61–70 of 206 posts
Re: PDF to Text, a challenging problem
#62Earlier quoted context omitted.
That's true, but it's dependent on the creator of the PDF having aligned incentives with the consumer of the PDF. In the e-Discovery field, it's commonplace for those providing evidence to dump it into a PDF purely so that it's harder for the opposing side's lawyers to consume. If both sides have lots of money this isn't a barrier, but for example public defenders don't have funds to hire someone (me!) to process the…
I wonder if AI will solve that
There are specialized models, but even generic ones like Gemini 2.0 Flash are really good and cheap, you can use them and embed the OCR inside the PDF to index to the original content.
Re: PDF to Text, a challenging problem
#63Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until something like this jars you out of that forgetfulness?
I definitely vaguely remember doing some incredibly cool things with PDFs and OCR about 6 or 7 years ago. Some project comes to mind... google tells me it was "tesseract" and that sounds familiar.
Re: PDF to Text, a challenging problem
#64I guess I'm lucky the PDF's I need to process are mostly rather dull unadventurous layouts. So far I've had great success using docling.
Re: PDF to Text, a challenging problem
#65For people who want people to read their documents[1] they should have their PDF point to a more digital-friendly format, an alt document.
Looks like you’ve found my PDF. You might want this version instead:
PDFs are often subpar. Just see the first example: standard Latex serif section title. I mean, PDFs often aren’t even well-typeset for what they are (dead-tree simulations).
[1] No sarcasm or truism. Some may just want to submit a paper to whatever publisher and go through their whole laundry list of what a paper ought to be. Wide dissemanation is not the point.
Re: PDF to Text, a challenging problem
#66Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…
[deleted]
Re: PDF to Text, a challenging problem
#67Earlier quoted context omitted.
Apart from lacking newlines, how is the result bad? It extracts the text for easy piping into an LLM.
- Most of the titles have incorrectly split words, for example "P ART 2—R EPEAL OF EPA R ULE R ELATING TO M ULTI -P OLLUTANT E MISSION S TANDARDS". I know LLMs are resilient against typos and mistakes like this, but it still seems not ideal. - The header is parsed in a way that I suspect would mislead an LLM: "BRETT GUTHRIE, KENTUCKY FRANK PALLONE, JR., NEW JERSEY CHAIRMAN RANKING MEMBER ONE HUNDRED NINETEENTH CONGRE…
> it seems to crash on any Arxiv PDF
Fixed, thanks for reporting :-)
Re: PDF to Text, a challenging problem
#68Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until someth…
No different than a fire ant whose leaf got knocked over by the wind and it moved on to the next.
Re: PDF to Text, a challenging problem
#69Below is a PDF. It is a .txt file. I can save it with a .pdf extension and open it in a PDF viewer. I can make changes in a text editor. For example, by editing this text file, I can change the text displayed on the screen when the PDF is opened, the font, font size, line spacing, the maximum characters per line, number of lines per page, the paper width and height, as well as portrait versus landscape mode.
%PDF-1.4
1 0 obj
>
endobj
2 0 obj
>
endobj
4 0 obj
>
endobj
5 0 obj
>
/ProcSet [ /PDF /Text ]
>>
endobj
6 0 obj
>
endobj
7 0 obj
>
stream
BT
/F1 50 Tf
1 0 0 1 50 752 Tm
54 TL
(PDF is)'
((a) a text format)'
((b) a graphics format)'
((c) (a) and (b).)'
()'
ET
endstream
endobj
8 0 obj
53
endobj
3 0 obj
>
endobj
xref
0 9
0000000000 65535 f Re: PDF to Text, a challenging problem
#70Below is a PDF. It is a .txt file. I can save it with a .pdf extension and open it in a PDF viewer. I can make changes in a text editor. For example, by editing this text file, I can change the text displayed on the screen when the PDF is opened, the font, font size, line spacing, the maximum characters per line, number of lines per page, the paper width and height, as well as portrait versus landscape mode. %PDF-1.4…
It can also have embedded binary streams. It was not made for text. It was made for layout and graphics. You give nice examples, but each of those lines could have been broken up into one call per character, or per word, even out of order.