Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

github.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#91

Post by smusamashah »

yigitkonur35 wrote: post #6
smusamashah wrote: post #4

I have not found any mention of accuracy. Since it's using LLM, how accurate the conversion is? As in does that NASA document match 100% with the pdf or did it introduce any made up things (hallucinations)? That converted NASA doc should be included in repo and linked in readme if you haven't already.

People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. The one in the repo right now is doing a pretty good job. Keep in mind though, this project is all about maxing out context for LLMs in products that need PDF input. We're not talking about some hardcore archiving system for the Library of Congress here. The goal is to boost consistency whenever you're feeding PDF…

> hallucinations, but you can totally tackle that with solid prompts.

This is so wrong. This so much sound as if you have not used LLMs to do any real work.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#92

Post by perrywky »

fkilaiwi wrote: post #53
pierre wrote: post #16

Parsing docs using LVM is the way forward (also see OCR2 paper released last week, people are having ablot of success parsing with fine tunned Qwen2). The hard part is to prevent the model ignoring some part of the page and halucinations (see some of the gpt4o sample here like the xanax notice: https://www.llamaindex.ai/blog/introducing-llamaparse-premiu... ) However this model will get better and we may soon have a…

what paper are you referring to?

I guess this: https://arxiv.org/html/2409.01704v1

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#93

Post by KoolKat23 »

This is handy, one thing I've noticed using 3.5 Sonnet, the tables that aren't the correct orientation are more prone to incorrect output.

I know this was an issue when GPT 4 vision initially came out due to training, not sure if it's a solved problem or if your tool handles this.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#94

Post by llm_trw »

Propelloni wrote: post #89
llm_trw wrote: post #48

Earlier quoted context omitted.

What's the total spend per quarter? For a margin that fat I'd be willing to jump through a lot of hoops if you're doing enough pages. The models (currently) fit in 24gb vram sequentially with small enough batch sizes, so a local server with consumer grade gpus wouldn't be impossible.

I'll check and get back to you. How can I reach you?

Email at omni_vision_ai@proton.me

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#95

Post by ithkuil »

whiplash451 wrote: post #79
yigitkonur35 wrote: post #61

Earlier quoted context omitted.

I get your worries about LLMs and their consistency problems. But I think we can fix a lot of that using LLMs themselves for checks. If you're after top-notch accuracy, you could throw in another prompt, add some visual and text input, and double-check that nothing's lost in translation. The cheaper models are actually great for this kind of quality control. LLMs have come a long way since they first showed up, and I…

If you can use an LLM for sanity checking, why can’t you use it for extraction at the first place?

Because currently models output a stream of tokens directly which are the performance and billing unit. Better models can do a better job at producing reasonable output but there is a limit to what can be done "on the fly".

Some models like openai o1 started employing internal "thinking" tokens which may or may not be equivalent to performing multiple passes with the same or different models but it has a similar effect.

One way to look at it is that if you want better results you have to put more computational resources in thinking. Also, just like humans, a team effort yields better results in producing well rounded results because you combine the strengths and you offset the weaknesses of different team members.

You can technically wrap all this into a single black box and have it converse with you as if it was one single entity that internally uses multiple models to think and cross check etc. The output is likely not going to be in real-time though and real time conversation was until now a very important feature.

In future we may on one hand relax the real time constraint and accept that for some tasks accuracy is more important than real time results.

Or we may eventually have faster machines or more clever algorithms that may "think" more in shorter amounts of time.

(Or a combination of the two)

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#96

Post by cyanydeez »

Oras wrote: post #56

While this is a nice development, it’s quite risky parsing documents with LLMs. In usual OCRs, you have boundaries to check, but with LLMs, you just get a black box output. As others mentioned, consistency is key in parsing documents and consistency is not a feature of LLMs. The output might look plausible, but without proper validation this is just a nice local playground that can’t make it to production.

Determinism is also up there because post processing can catch and fix common errors
Post reply on HN