I've also been using the nougat models from meta, which are trained to turn PDF into md using the donut architecture
Does it work well on documents that aren't academic papers?
Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
51–60 of 97 posts
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#52I had been using GPT4o for extracting insights from Scanned docs, it was doing fine. But very recently (since they launched new model - o1), it's not working. GPT4o is refusing to extract text from images and says it can't do it, though it was doing same thing with same prompts till last week. I am not sure if this is intentional downgrade and it can be clubbed with new model launch, but it's really frustrating for m…
Have you tried few shot prompting? Something on the lines of:
User: Extract x from the given scanned document.
Assistant:
User: Extract x from the given scanned document.
Assistant:
User: Extract x from the given scanned document.
In my experience, this seems to make the model significantly more consistent.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#53Parsing docs using LVM is the way forward (also see OCR2 paper released last week, people are having ablot of success parsing with fine tunned Qwen2). The hard part is to prevent the model ignoring some part of the page and halucinations (see some of the gpt4o sample here like the xanax notice: https://www.llamaindex.ai/blog/introducing-llamaparse-premiu... ) However this model will get better and we may soon have a…
what paper are you referring to?
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#54I had been using GPT4o for extracting insights from Scanned docs, it was doing fine. But very recently (since they launched new model - o1), it's not working. GPT4o is refusing to extract text from images and says it can't do it, though it was doing same thing with same prompts till last week. I am not sure if this is intentional downgrade and it can be clubbed with new model launch, but it's really frustrating for m…
This. Inconsistency is a big problem for large tasks, you are better off making your own models to do this. I have seen this odd kind of inconsistency in generating the same results, sometimes in the same chat itself after starting off fine. I was once trying to extract hand written dates and times from a large pdf document in batches of 10 pages at a time from a very specific part of the page. IN some documents it s…
> "...you are better off making your own models to do this"
I'm doubtful you meant what you wrote here. Using a readymade UI or API to perform an effectively magical task (for most of us) is an entirely different paradigm to "just train your own model."
In reality, for us non-ML model training mortals, we're actually probably better off hiring a human to do basic data entry.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#55Does it do image to MD too?
[deleted]
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#56While this is a nice development, it’s quite risky parsing documents with LLMs.
In usual OCRs, you have boundaries to check, but with LLMs, you just get a black box output.
As others mentioned, consistency is key in parsing documents and consistency is not a feature of LLMs.
The output might look plausible, but without proper validation this is just a nice local playground that can’t make it to production.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#57Parsing docs using LVM is the way forward (also see OCR2 paper released last week, people are having ablot of success parsing with fine tunned Qwen2). The hard part is to prevent the model ignoring some part of the page and halucinations (see some of the gpt4o sample here like the xanax notice: https://www.llamaindex.ai/blog/introducing-llamaparse-premiu... ) However this model will get better and we may soon have a…
We’ve been doing exactly this by doubling-down on VLMs (https://vlm.run)
- VLMs are way better at handling layout and context where OCR systems fail miserably
- VLMs read documents like humans do, which makes dealing with special layouts like bullets, tables, charts, footnotes much more tractable with a singular approach rather than have to special case a whole bunch of OCR + post-processing
- VLMs are definitely more expensive, but can be specialized and distilled for accurate and cost effective inference
In general, I think vision + LLMs can be trained to explicitly to “extract” information and avoid reasoning/hallucinating about the text. The reasoning can be another module altogether.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#58GPT 4o doesn't do actual OCR and there's much smaller and more effective models for specifically this problem.
I appreciate your work, intent, and sharing it. It's very important to appreciate what you're doing and its context when sharing it.
At that point, you are responsible for it, and the choices you make when communicating about it reflect on you.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#59Was just looking for something like this. Does it handle equations to latex or similar? How about rotated tables, ie landscape mode but page is still portait?
Check out Nougat from meta
Thanks, looks very interesting, but also somewhat abandoned. Will keep an eye on it in case someone picks up the torch.
Re: Show HN: PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o
#60While this is a nice development, it’s quite risky parsing documents with LLMs. In usual OCRs, you have boundaries to check, but with LLMs, you just get a black box output. As others mentioned, consistency is key in parsing documents and consistency is not a feature of LLMs. The output might look plausible, but without proper validation this is just a nice local playground that can’t make it to production.
Speaking of the devil - I've just had hallucinations with Ollama and reader-lm model (converting HTML to Markdown) the other day. In 40% of cases it spew out things that weren't in the input (not exactly surprising, knowing that it's a generative model).
Turns out the model needs temperature of zero (and then it seem to behave well, at least in simple tests), but it wasn't in the model settings.
https://github.com/ollama/ollama/issues/6875#issuecomment-23...