Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama-Scan: Convert PDFs to Text W Local LLMs

github.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#61

Post by threeducks »

nawazgafar wrote: post #24
fcoury wrote: post #22

I really wanted this to be good. Unfortunately it converted a page that contained a table that is usually very hard for converters to properly convert and I got a full page with "! Picture 1:" and nothing else. On top of that, it hung at page 17 of a 25 page document and never resumed.

Author here, that sucks. I'd love to recreate this locally. Would you be willing to share the PDF?

As far as I am aware, the "hanging" issue remains unsolved to this day. The underlying problem is that LLMs sometimes get stuck in a loop where they repeat the same text again and again until they reach the token limit. You can break the loop by setting a repeat penalty, but when your image contains repeated text, such as in tables, the LLM will output incorrect results to prevent repetition.

Here is the corresponding GitHub issue for your default model (Qwen2.5-VL):

https://github.com/QwenLM/Qwen2.5-VL/issues/241

You can mitigate the fallout of this repetition issue to some degree by chopping up each page into smaller pieces (paragraphs, tables, images, etc.) with a page layout model. Then at least only part of the text is broken instead of the entire page.

A better solution might be to train a model to estimate a heat map of character density for a page of text. Then, condition the vision-language model on character density by feeding the density to the vision encoder. Also output character coordinates, which can be used with the heat map to adjust token probabilities.

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#62

Post by davidwritesbugs »

deepsquirrelnet wrote: post #44

Give the nanonets-ocr-s model a try. It’s a fine tune of Qwen 2.5 vl which I’ve had good success with for markdown and latex with image captioning. It uses a simple tagging scheme for page numbers, captions and tables.

I've tried nanonets but it seems very sensitive to the prompt, changing it slightly turned the output to rubbish. When it worked it was pretty good.

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#63

Post by worldsayshi »

visarga wrote: post #58

The crucial information is missing - accuracy comparison with other OCR providers. From my experience LLM based OCR might misread the layout and hallucinate values, it is very subtle but sometimes critically wrong. Classical OCR has more precision but doesn't get the layout at all. Combining both has other issues, no approach is 100% reliable.

Yes I tried using LLM for reading CV:s a while back and I really struggled with getting it to not omit important information.

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#64

Post by cess11 »

firesteelrain wrote: post #4
david_draco wrote: post #3

Looking at the code, this converts PDF pages to images, then transcribes each image. I might have expected a pdftotext post-processor. The complexity of PDF I guess ...

There is a very popular Python module called ocrmypdf. I used it to help my HOA and OCR’ing of old PDFs. https://github.com/ocrmypdf/OCRmyPDF No LLMs required.

It's nice, I've used it as a fallback text extraction method in an ETL flow that chugged through tens of thousands of corporate and legal PDF files.

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#65

Post by __alexs »

Y_Y wrote: post #37
HocusLocus wrote: post #11

By 1990 Omnipage 3 and its successors were 'good enough' and with their compact dictionaries and letter form recognition were miracles of their time at ~300MB installed. In 2025 LLMs can 'fake it' using Trilobites of memory and Petaflops. It's funny actually, like a supercomputer being emulated in real time on a really fast Jacquard loom. By 2027 even simple hand held calculator addition will be billed in kilowatt-ho…

https://en.wikipedia.org/wiki/Trilobite Trilobites? Those were truly primitve computers.

Didn't the discworld books have these?

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#66

Post by bayindirh »

privatelypublic wrote: post #38
HocusLocus wrote: post #11

By 1990 Omnipage 3 and its successors were 'good enough' and with their compact dictionaries and letter form recognition were miracles of their time at ~300MB installed. In 2025 LLMs can 'fake it' using Trilobites of memory and Petaflops. It's funny actually, like a supercomputer being emulated in real time on a really fast Jacquard loom. By 2027 even simple hand held calculator addition will be billed in kilowatt-ho…

If you think 1990's ocr- even 2000's OCR is remotely as good as modern OCR... I`v3 g0ta bnedge to sell.

Tesseract can do wonders for scanned paper (and web generated PDFs) both in its old and new version. If you want to pay for something closed, Prizmo on macOS is extremely good as well.

On the other hând, LLm5 are sl0wwer, moré resource hangry and l3ss accurale fr their outpu1z.

We shoulD stop gl0rıfying LLMs for 3verylhin9.

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#69

Post by agentcoops »

visarga wrote: post #58

The crucial information is missing - accuracy comparison with other OCR providers. From my experience LLM based OCR might misread the layout and hallucinate values, it is very subtle but sometimes critically wrong. Classical OCR has more precision but doesn't get the layout at all. Combining both has other issues, no approach is 100% reliable.

Have you evaluated this lately? Last year or even just earlier this year I would have mostly agreed with you. At this point, however, with at least the documents I have been working on, OCR reliability with GPT5 or Mistral OCR [1] has been much better than even domain-trained classical OCR. If the documents have even slightly complex layout (to say nothing of page numbers or page headings or an uncommon font), the accuracy of state of the art LLMs has been in my work an order of magnitude greater. The ability to have the LLM tentatively combine trailing sentences across pages, which is especially useful if you have to work with documents in German say, is invaluable.

[1] https://mistral.ai/news/mistral-ocr

Re: Llama-Scan: Convert PDFs to Text W Local LLMs

#70

Post by agentcoops »

bayindirh wrote: post #66
privatelypublic wrote: post #38

Earlier quoted context omitted.

If you think 1990's ocr- even 2000's OCR is remotely as good as modern OCR... I`v3 g0ta bnedge to sell.

Tesseract can do wonders for scanned paper (and web generated PDFs) both in its old and new version. If you want to pay for something closed, Prizmo on macOS is extremely good as well. On the other hând, LLm5 are sl0wwer, moré resource hangry and l3ss accurale fr their outpu1z. We shoulD stop gl0rıfying LLMs for 3verylhin9.

I've worked extensively with Tesseract, ABBYY, etc in a personal and professional context. Of course they work well for English-language documents without any complexity of layout that are scanned without the slightest defect. At this point, based on extensive testing for work, state of the art LLMs simply have better accuracy -- and an order of magnitude so if you have non-English documents with complex layouts and less than ideal scans. I'll give you speed, but the accuracy is so much greater (and the need for human intervention so much less) that in my experience it's a worthwhile trade-off.

I'm not saying this applies to you, but my sense from this thread is that many are comparing the results of tossing an image into a free ChatGPT session with an "OCR this document" prompt to a competent Tesseract-based tool... LLMs certainly don't solve any and every problem, but this should be based on real experiments. In fact, OCR is probably the main area where I've found them to simply be the best solution for a professional system.

Post reply on HN