Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

github.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#21

Post by itissid »

zerop wrote: post #18

I had been using GPT4o for extracting insights from Scanned docs, it was doing fine. But very recently (since they launched new model - o1), it's not working. GPT4o is refusing to extract text from images and says it can't do it, though it was doing same thing with same prompts till last week. I am not sure if this is intentional downgrade and it can be clubbed with new model launch, but it's really frustrating for m…

This. Inconsistency is a big problem for large tasks, you are better off making your own models to do this.

I have seen this odd kind of inconsistency in generating the same results, sometimes in the same chat itself after starting off fine.

I was once trying to extract hand written dates and times from a large pdf document in batches of 10 pages at a time from a very specific part of the page. IN some documents it started by refusing, but not in other different chat windows that I tried with the same document. Sometimes it would say there is an error, and then it would work in a new chat window. But I am not sure why, but just starting a new chat works for these kind of situations.

Sometimes it will start off fine with OCR, then as the task progresses, it will start hallucinating. Even though the text to be extracted follows a pattern like dates, it for the life of me could not get it right.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#22

Post by freedomben »

yigitkonur35 wrote: post #6
smusamashah wrote: post #4

I have not found any mention of accuracy. Since it's using LLM, how accurate the conversion is? As in does that NASA document match 100% with the pdf or did it introduce any made up things (hallucinations)? That converted NASA doc should be included in repo and linked in readme if you haven't already.

People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. The one in the repo right now is doing a pretty good job. Keep in mind though, this project is all about maxing out context for LLMs in products that need PDF input. We're not talking about some hardcore archiving system for the Library of Congress here. The goal is to boost consistency whenever you're feeding PDF…

Can you give some examples of prompts that you use that will tackle hallucinations?

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#23

Post by authorfly »

yigitkonur35 wrote: post #9
johndough wrote: post #8

Earlier quoted context omitted.

> I think it offers an optimal balance of affordability & reliability. It is hard to trust "you" when ChatGPT wrote that text. You never know which part of the answer is genuine and which part was made up by ChatGPT. To actually answer that question: Pricing varies quite a bit depending on what exactly you want to do with a document. Text detection generally costs $1.5 per 1k pages: https://cloud.google.com/vision/pr…

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

I will say I have had a look at your code here. I really do value your innovation here in gaining better accuracy, but I don't think it's is much more accurate for obscure PDF cases - Maybe it halves those obscure errors. I found it still hallucinated or failed to parse some text (e.g. that unusual languages, screenshots with tiny blurred JPEG text, images/shapes remain hallucination issues with your solution). BTW I noticed a small typo "Convert document as is be creative to use markdown effectively" in the prompt. For me changing this and adding text about returning "None" if the text is unreadable reduced hallucinations.

Would you contrast your accuracy with Textract? Because Textract is 10x cheaper than this at approx 1 cent per page (and 20x cheaper than Cloudconvert). What documents make more sense to use with your tool? Is it worth waiting till gpt-4o costs drop 10x with the same quality level (i.e. not gpt-4o-mini) to use this? In my use case it's better to drop than to hallucinate.

What do you think makes sense in relation to Textract?

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#24

Post by authorfly »

pierre wrote: post #16

Parsing docs using LVM is the way forward (also see OCR2 paper released last week, people are having ablot of success parsing with fine tunned Qwen2). The hard part is to prevent the model ignoring some part of the page and halucinations (see some of the gpt4o sample here like the xanax notice: https://www.llamaindex.ai/blog/introducing-llamaparse-premiu... ) However this model will get better and we may soon have a…

What about combining old school OCR with GPT visual OCR?

If your old school OCR output has output that is not present in the visual one, but is coherent (e.g. english sentences), you could get it back and slot it into the missing place from the visual output.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#25

Post by hnlmorg »

yigitkonur35 wrote: post #9
johndough wrote: post #8

Earlier quoted context omitted.

> I think it offers an optimal balance of affordability & reliability. It is hard to trust "you" when ChatGPT wrote that text. You never know which part of the answer is genuine and which part was made up by ChatGPT. To actually answer that question: Pricing varies quite a bit depending on what exactly you want to do with a document. Text detection generally costs $1.5 per 1k pages: https://cloud.google.com/vision/pr…

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

AWS Textract does use ML and I’ve personally used it to parse tables for automated invoice processing.

You wouldn’t get a markdown document automatically generated (or at least you couldn’t when I last used it a few years ago) but you did get an XML document

That XML document was actually better for our purposes because it gives you a confidence score and is properly structured, so floating frame, tables and columns would be properly structured in the output document. This reduces the risk of hallucinations.

It’s less of an out-of-the-box solution but that’s to be expected with AWS APIs.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#27

Post by llm_trw »

yigitkonur35 wrote: post #9
johndough wrote: post #8

Earlier quoted context omitted.

> I think it offers an optimal balance of affordability & reliability. It is hard to trust "you" when ChatGPT wrote that text. You never know which part of the answer is genuine and which part was made up by ChatGPT. To actually answer that question: Pricing varies quite a bit depending on what exactly you want to do with a document. Text detection generally costs $1.5 per 1k pages: https://cloud.google.com/vision/pr…

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

That is not a tricky example. Those tables are as clear cut as clear cut can be.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#28

Post by llm_trw »

Propelloni wrote: post #15
yigitkonur35 wrote: post #9

Earlier quoted context omitted.

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

> you might get some skipped bits, but that's not a dealbreaker for folks looking to feed PDFs into AI systems Unless it is. We have a few hundred PDF per month (mostly tables) where we need 100% accuracy. Currently we feed them into an OCR and have humans check the result. I do not win anything if I have to check the LLM output, too.

I'm currently solving this problem for work and thinking of a spin out, what's a ballpark figure you'd be willing to pay per 1000 pages for 99.999% character level accuracy?

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#29

Post by eth0up »

I used GPT4o to convert heavily convoluted PDFs into csv files. The files were Florida Lottery Pick(n) histories, which they deliberately convolute to prevent automatic searching; ctrl-f does nothing and a fsck-ton of special characters embellish the whole file.

I had previously done so manually, with regex, and was surprised with the quality of the end results of GPT, despite many preceding failed iterations. The work was done in two steps, first with pdf2text, then python.

I'm still trying to created a script to extract the latest numbers from the FL website and append to a cvs list, without re-running the stripping script on the whole PDF every time. Why? I want people to have the ability to freely search the entire history of winning numbers, which in their web hosted search function, is limited to only two of 30+ years.

I know there's a more efficient method, but I don't know more than that.

Post reply on HN