Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenDataLoader-PDF: An open source tool for structured PDF parsing

github.com

Post reply on HN
Print view View original

1–10 of 31 posts

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#3

Post by simonw »

clueless wrote: post #2

Given the current llm context size limitation, what is the state of art for feeding large doc/text blobs into llm for accurate processing?

The current generation of models all support pretty long context now - the Gemini family has had 1m tokens for over a year, GPT-4.1 is 1m, interestingly GPT-5 is back down to 400,000, Claude 4 is 200,000 but there's a mode of Claude Sonnet 4 that can do 1m as well.

The bigger question is how well they perform - there are needle-in-haystack benchmarks that test that, they're mostly scoring quite highly on those now.

https://cloud.google.com/blog/products/ai-machine-learning/t... talks about that for Gemini 1.5.

Here's a couple of relevant leaderboards: https://huggingface.co/spaces/RMT-team/babilong and https://longbench2.github.io/

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#4

Post by trevor-e »

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious.

I've been toying with an idea of a new format that stores text naturally and captures semantics (e.g. to help with table parsing), but also preserves formatting rules so you can still achieve fairly consistent rendering. This format could be easily converted to PDF, although the opposite conversion would have the regular challenges. The main challenge is distribution of course.

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#5

Post by lysecret »

clueless wrote: post #2

Given the current llm context size limitation, what is the state of art for feeding large doc/text blobs into llm for accurate processing?

Generally use 2.5 flash for this, works incredibly well. So many traditionally hard things can now we solved by stuffing it into a pretty cheap llm haha.

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#6

Post by clueless »

simonw wrote: post #3
clueless wrote: post #2

Given the current llm context size limitation, what is the state of art for feeding large doc/text blobs into llm for accurate processing?

The current generation of models all support pretty long context now - the Gemini family has had 1m tokens for over a year, GPT-4.1 is 1m, interestingly GPT-5 is back down to 400,000, Claude 4 is 200,000 but there's a mode of Claude Sonnet 4 that can do 1m as well. The bigger question is how well they perform - there are needle-in-haystack benchmarks that test that, they're mostly scoring quite highly on those now. h…

sorry I should have been more clear, I meant around open source llms. and I guess the question is, how are closed source llm doing it so well. And if OS OpenNote is the best we have...

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#8

Post by Jaxan »

trevor-e wrote: post #4

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious. I'…

Wouldn’t it be better to invest in a human-friendly format first (which also could be AI-friendly).

Re: OpenDataLoader-PDF: An open source tool for structured PDF parsing

#10

Post by trevor-e »

Jaxan wrote: post #8
trevor-e wrote: post #4

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious. I'…

Wouldn’t it be better to invest in a human-friendly format first (which also could be AI-friendly).

Not really sure what you mean by a "human-friendly" file format, can you elaborate? File formats are inherently not friendly to humans, they are a bag of bytes. But that doesn't mean they can't be better consumed by tools which is what I mean by "AI friendly".
Post reply on HN