Live data from Hacker News

OpenDataLoader-PDF: An open source tool for structured PDF parsing

github.com

11–20 of 31 posts

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#11
I just tested it on one of my nemeses: PDF bank statements. They're surprisingly tough to work with if you want to get clean, structured transaction data out of them.

The JSON extract actually looks pretty good and seems to produce something usable in one shot, which is very good compared to all the other tools I've tried so far, but I still need to check it more in-depth.

Sharing here in case someone chimes in with "hey, doofus, $magic_project already solves this."

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#12
post #5
post #2

Given the current llm context size limitation, what is the state of art for feeding large doc/text blobs into llm for accurate processing?

Generally use 2.5 flash for this, works incredibly well. So many traditionally hard things can now we solved by stuffing it into a pretty cheap llm haha.

What do you mean by “traditionally hard” in relation to a pdf? Most if not all of the docs I’m tasked with parsing are secured, flattened, and handwritten, which can cause any tool (traditional or ai) to require a confidence score and manual intervention. Also might be that i just get stuck with the edge cases 90% of the time.

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#14

I just tested it on one of my nemeses: PDF bank statements. They're surprisingly tough to work with if you want to get clean, structured transaction data out of them. The JSON extract actually looks pretty good and seems to produce something usable in one shot, which is very good compared to all the other tools I've tried so far, but I still need to check it more in-depth. Sharing here in case someone chimes in with…

For 'zoned' extraction, Cermine[0] may be of use as a pre-processing step. Mileage may vary as its tailored towards papers.

[0]: http://cermine.ceon.pl/about.html

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#15
post #8
post #4

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious. I'…

Wouldn’t it be better to invest in a human-friendly format first (which also could be AI-friendly).

If you can convince your bank to make available your bank statement in Markdown, let us know.

Your transactions are probably already available in CSV.

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#16

I just tested it on one of my nemeses: PDF bank statements. They're surprisingly tough to work with if you want to get clean, structured transaction data out of them. The JSON extract actually looks pretty good and seems to produce something usable in one shot, which is very good compared to all the other tools I've tried so far, but I still need to check it more in-depth. Sharing here in case someone chimes in with…

Camelot[1] worked very well for me with bank statements. Disclaimer: I'm one of the core contributors.

[1] https://github.com/camelot-dev/camelot

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#19
post #9

How does it compare to docling?

Docling primarily uses AI models to extract PDF content, this project looks like it uses a custom parser written in Java, built atop veraPDF.

Correct me if I am wrong, but Docling can do both. It has also, among other strategies, a non-AI pipeline to determine the layout (based on qpdf I believe). So these projects are not that different.

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#20
post #4

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious. I'…

Doesn't Latex do this?
Post reply on HN