Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama-OCR: Document to Markdown

llamaocr.com

Post reply on HN
Print view View original

11–20 of 104 posts

Re: Llama-OCR: Document to Markdown

#11

Post by M4v3R »

Eisenstein wrote: post #8

All it does is send the image to Llama 3.2 Vision and ask for it to read the text. Note that this is just as open to hallucination as any other LLM output, because what it is doing is not reading the pixels looking for text characters, but describing the picture, which uses the images it trained on and their captions to determine what the text is. It may completely make up words, especially if it can't read them.

This is also true for any other OCR system, we just never called these errors “hallucinations” in this context.

Re: Llama-OCR: Document to Markdown

#13

Post by llm_trw »

M4v3R wrote: post #11
Eisenstein wrote: post #8

All it does is send the image to Llama 3.2 Vision and ask for it to read the text. Note that this is just as open to hallucination as any other LLM output, because what it is doing is not reading the pixels looking for text characters, but describing the picture, which uses the images it trained on and their captions to determine what the text is. It may completely make up words, especially if it can't read them.

This is also true for any other OCR system, we just never called these errors “hallucinations” in this context.

It really isn't since those systems are character based.

Re: Llama-OCR: Document to Markdown

#14

Post by rajansheth »

bbor wrote: post #2

Looks awesome! Been doing a lot of OCR recently, and love the addition to the space. The reigning champion in the PDF -> Markdown space (AFAIK) is Facebook's Nougat[1], and I'm excited to hook this up to DSPy and see which works better for philosophy books. This repo links the Zerox[2] project by some startup, which also looks awesome , and certainly more smoothly advertised than Nougat. Would love corrections/advice…

together.ai serves 100+ open-source models including multi-modal Llama 3.2 with an OpenAI compatible API

Re: Llama-OCR: Document to Markdown

#17

Post by notsylver »

I've been doing a lot of OCR recently, mostly digitising text from family photos. Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. It still had enough failures and hallucinations to make it faster to write it in by hand. Annoying considering how close it feels to working.

This seems worse. Sometimes it replies with just the text, sometimes it replies with a full "The image is a scanned document with handwritten text...". I was hoping for some fine tuning or something for it to beat Gemini Flash, it would save me a lot of time. :(

Re: Llama-OCR: Document to Markdown

#18

Post by famouswaffles »

notsylver wrote: post #17

I've been doing a lot of OCR recently, mostly digitising text from family photos. Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. It still had enough failures and hallucinations to make it faster to write it in by hand. Annoying considering how close it feels to working. This seems worse. Sometimes it replies with just the text,…

>Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close.

For Normal models, the state of Open Source OCR is pretty terrible. Unfortunately, the closed options from Microsoft, Google etc are much better. Did you try those ?

Interesting about Flash, what LLMs did you test ?

Re: Llama-OCR: Document to Markdown

#19

Post by geysersam »

M4v3R wrote: post #11
Eisenstein wrote: post #8

All it does is send the image to Llama 3.2 Vision and ask for it to read the text. Note that this is just as open to hallucination as any other LLM output, because what it is doing is not reading the pixels looking for text characters, but describing the picture, which uses the images it trained on and their captions to determine what the text is. It may completely make up words, especially if it can't read them.

This is also true for any other OCR system, we just never called these errors “hallucinations” in this context.

I gave this tool a picture of a restaurant menu and it made up several additional entries that didn't exist in the picture... What other OCR system would do that?
Post reply on HN