Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

github.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#71

Post by yigitkonur35 »

llm_trw wrote: post #38
Lerc wrote: post #35

Earlier quoted context omitted.

I guess it depends on the use case, but if it surpasses the error rate that exists in the source document then it would be difficult to argue against. Specific things like evidentiary use would want 100% but that's at a level where any document processing would be suspect. What is the the typical range for error rate in PDF generation in various fields? Even robust technical documents have the occasional typo.

I'm not using generative models to fill in details not present in the original document. If there's a typo there then there will be a typo in the transcript. If you want to fix that then you can run another model on top of it.

You're right. For my API that prepares PDFs for LLMs, fixing typos makes sense. But yeah, keeping original text is crucial for most OCR tasks.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#72

Post by yigitkonur35 »

eth0up wrote: post #29

I used GPT4o to convert heavily convoluted PDFs into csv files. The files were Florida Lottery Pick(n) histories, which they deliberately convolute to prevent automatic searching; ctrl-f does nothing and a fsck-ton of special characters embellish the whole file. I had previously done so manually, with regex, and was surprised with the quality of the end results of GPT, despite many preceding failed iterations. The wo…

I really appreciate you sharing your hands-on experience with a real-world scenario. It's interesting how people unfamiliar with traditional OCR often doubt LLMs, but having worked with actual documents, I know how inefficient classic OCR methods can be. So these minor errors don't alarm me at all. Your use case sounds fascinating - I might just incorporate it into my own benchmarks. Thanks again for your insightful comment!

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#73

Post by [deleted] »

alchemist1e9 wrote: post #31
eth0up wrote: post #29

I used GPT4o to convert heavily convoluted PDFs into csv files. The files were Florida Lottery Pick(n) histories, which they deliberately convolute to prevent automatic searching; ctrl-f does nothing and a fsck-ton of special characters embellish the whole file. I had previously done so manually, with regex, and was surprised with the quality of the end results of GPT, despite many preceding failed iterations. The wo…

Off topic - but the obvious follow up question is why do you want people to have this ability to search the entire history?

[deleted]

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#76

Post by vunderba »

scottmcdot wrote: post #30

Does it do image to MD too?

Unless the only thing you want is a description of the image, then the real answer is NO. You can get an LLM to do something like "If you encounter an image that is not easily convertable to standard markdown, insert a [[DESCRIPTION OF IMAGE]] here." placeholder, but at that point you've lost information that may be salient to the original PDF.

The reason is because these multimodal LLMs can give you descriptions/OCR/etc., but they cannot give you quantifiable information related to placement.

So if there was a picture of a tiger in the middle of the page converted to a bitmap, you couldn't get the LLM to give you something like this: "Image detected at pixel position (120, 200) - (240, 500)." - because that's really want you want.

You almost need segmentation system middleware that the LLM can forward to which can cut out these images to use in markdown syntax:

    ![A tiger burning brightly](/assets/images/tiger.png)

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#77

Post by refulgentis »

yigitkonur35 wrote: post #9
johndough wrote: post #8

Earlier quoted context omitted.

> I think it offers an optimal balance of affordability & reliability. It is hard to trust "you" when ChatGPT wrote that text. You never know which part of the answer is genuine and which part was made up by ChatGPT. To actually answer that question: Pricing varies quite a bit depending on what exactly you want to do with a document. Text detection generally costs $1.5 per 1k pages: https://cloud.google.com/vision/pr…

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

You and your delegated writing are lying about cost.

It is off by 2 orders of magnitude.

My guess is you're using the token counting algorithm for pre-4o with the costs for 4o and later.

That aside, I strongly suggest taking a week off from code-outside-work and use that time to reflect-as-work. The post and ensuing comments are a horror show. Don't take it too hard, it probably won't matter in the long run, no ones going to remember.

But you'd get a lot out of taking it harder than you did in the comments I've seen, including one this morning where you replied to me. It worries me that you don't seem to understand how sloppy this work is.

When I was 14, my math teacher gave me a 0 on a test because I just wrote the answers instead of showing work. That gave me a powerful appreciation for being precise, clear, and accurate.

The only positive outcome is that even though there was enough upvotes for a simple, sloppy, mispurposed GPT wrapper to end up on the front page for ~16 hours, near-universally, the comments seem to understand contextually there's a lot of problems with how this was shared.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#78

Post by malcolmhere »

hnlmorg wrote: post #25
yigitkonur35 wrote: post #9

Earlier quoted context omitted.

You've got a point, but try testing it on a tricky example like the Apollo 17 document - you know, with those sideways tables and old-school writing. You'll see all three non-AI services totally bomb. Now, if you tweak it to batch = 1 instead of 10, you'll notice there's hardly any made-up stuff. When you dial down the temperature close to zero, it's super unlikely to see hallucinations with limited context. At worst…

AWS Textract does use ML and I’ve personally used it to parse tables for automated invoice processing. You wouldn’t get a markdown document automatically generated (or at least you couldn’t when I last used it a few years ago) but you did get an XML document That XML document was actually better for our purposes because it gives you a confidence score and is properly structured, so floating frame, tables and columns…

You can get Markdown nowadays too, at least using this Python wrapper:

https://aws-samples.github.io/amazon-textract-textractor/not...

It's very consistent, though pricey.

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#79

Post by whiplash451 »

yigitkonur35 wrote: post #61
Oras wrote: post #56

While this is a nice development, it’s quite risky parsing documents with LLMs. In usual OCRs, you have boundaries to check, but with LLMs, you just get a black box output. As others mentioned, consistency is key in parsing documents and consistency is not a feature of LLMs. The output might look plausible, but without proper validation this is just a nice local playground that can’t make it to production.

I get your worries about LLMs and their consistency problems. But I think we can fix a lot of that using LLMs themselves for checks. If you're after top-notch accuracy, you could throw in another prompt, add some visual and text input, and double-check that nothing's lost in translation. The cheaper models are actually great for this kind of quality control. LLMs have come a long way since they first showed up, and I…

If you can use an LLM for sanity checking, why can’t you use it for extraction at the first place?

Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o

#80

Post by fzysingularity »

yigitkonur35 wrote: post #63
fzysingularity wrote: post #57

Earlier quoted context omitted.

We’ve been doing exactly this by doubling-down on VLMs ( https://vlm.run ) - VLMs are way better at handling layout and context where OCR systems fail miserably - VLMs read documents like humans do, which makes dealing with special layouts like bullets, tables, charts, footnotes much more tractable with a singular approach rather than have to special case a whole bunch of OCR + post-processing - VLMs are definitely m…

I did a ton of Googling before writing this code, but I couldn't find you guys anywhere. If I had, I'd have definitely used your stuff. You might want to think about running some small-scale Google Ads campaigns. They could be especially effective if you target people searching for both LLM and OCR together. Great product, congratz!

Hey, thanks! DM me if you want to test it out (sudeep@vlm.run).

Agreed on SEO - we’re redoing our landing page and searchability. We recently rebranded, hence the lack of direct search hits for LLM / OCR.

Post reply on HN