Holy Hallucinations batman!
Even the example images hallucinates random text
Llama-OCR: Document to Markdown
31–40 of 104 posts
Re: Llama-OCR: Document to Markdown
#32I've been doing a lot of OCR recently, mostly digitising text from family photos. Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. It still had enough failures and hallucinations to make it faster to write it in by hand. Annoying considering how close it feels to working. This seems worse. Sometimes it replies with just the text,…
That's a bummer. I'm trying to do the exact same thing right now, digitize family photos. Some of mine have German on the back. The last OCR to hit headlines was terrible, was hoping this would be better. ChatGPT 4o has been good though, when I paste individual images into the chat. I haven't tried with the API yet, not sure how much that would cost me to process 6500 photos, many of which are blank but I don't have…
Use a local rubbish model to extract text. If it doesn’t find any on the back, don’t send it to chatgtp?
Terrascan comes to mind
Re: Llama-OCR: Document to Markdown
#33All it does is send the image to Llama 3.2 Vision and ask for it to read the text. Note that this is just as open to hallucination as any other LLM output, because what it is doing is not reading the pixels looking for text characters, but describing the picture, which uses the images it trained on and their captions to determine what the text is. It may completely make up words, especially if it can't read them.
This is also true for any other OCR system, we just never called these errors “hallucinations” in this context.
No, it's not even close to OCR systems, which are based on analyzing points in a grid for each character stroke and comparing them with known characters. Just for one thing, OCR systems are deterministic. Deterministic. Look it up.
Re: Llama-OCR: Document to Markdown
#34I gave it a sentence, which I created by placing 500 circles via a genetic algorithm to form a sentence. And then drew with an actual physical circle:
https://www.instagram.com/marekgibney/p/BiFNyYBhvGr/
Interestingly, it sees the circles just fine, but not the sentence. It replied with this:
The image contains no text or other elements
that can be represented in Markdown. It is a
visual composition of circles and does not
convey any information that can be translated
into Markdown format.Re: Llama-OCR: Document to Markdown
#35Holy Hallucinations batman! Even the example images hallucinates random text
Same for me. The receipt headline only says "Trader Joe's" and yet the model insists on adding some information and transcribes "Trader Joe's Receipt". This is like Xeroxgate, but infinitely worse.
Someday this will do great damage in ways we will completely neglect and overlook.
Re: Llama-OCR: Document to Markdown
#36I gave it a sentence, which I created by placing 500 circles via a genetic algorithm to form a sentence. And then drew with an actual physical circle: https://www.instagram.com/marekgibney/p/BiFNyYBhvGr/ Interestingly, it sees the circles just fine, but not the sentence. It replied with this: The image contains no text or other elements that can be represented in Markdown. It is a visual composition of circles and do…
I can’t read anything but the „stop“ either without seeing the solution first
Re: Llama-OCR: Document to Markdown
#37I gave it a sentence, which I created by placing 500 circles via a genetic algorithm to form a sentence. And then drew with an actual physical circle: https://www.instagram.com/marekgibney/p/BiFNyYBhvGr/ Interestingly, it sees the circles just fine, but not the sentence. It replied with this: The image contains no text or other elements that can be represented in Markdown. It is a visual composition of circles and do…
I can't read this either.
Edit: at a distance it's easier to read
Re: Llama-OCR: Document to Markdown
#38I gave it a sentence, which I created by placing 500 circles via a genetic algorithm to form a sentence. And then drew with an actual physical circle: https://www.instagram.com/marekgibney/p/BiFNyYBhvGr/ Interestingly, it sees the circles just fine, but not the sentence. It replied with this: The image contains no text or other elements that can be represented in Markdown. It is a visual composition of circles and do…
Why is it interesting? The image does not look like anything, and you need to skew it (by looking at an angle) to see any letters (barely).
Re: Llama-OCR: Document to Markdown
#39Earlier quoted context omitted.
This is also true for any other OCR system, we just never called these errors “hallucinations” in this context.
No, it's not even close to OCR systems, which are based on analyzing points in a grid for each character stroke and comparing them with known characters. Just for one thing, OCR systems are deterministic. Deterministic . Look it up.
OCR system use vision models and as such they can make mistakes. They don't sample but they produce a distribution of probability over words like LLMs.
Re: Llama-OCR: Document to Markdown
#40I've been doing a lot of OCR recently, mostly digitising text from family photos. Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. It still had enough failures and hallucinations to make it faster to write it in by hand. Annoying considering how close it feels to working. This seems worse. Sometimes it replies with just the text,…
Have you tried downscaling the images? I started getting better results with lower resolution images. I was using scans made with mobile phone cameras for this. convert -density 76 input.pdf output-%d.png https://github.com/philips/paper-bidsheets
That's interesting. I downscaled the images to something like 800px but that was mostly to try improve upload times. I wonder if downscaling further and with a better algorithm would help.. I remember using CLIP and found different scaling algorithms helped text readability. Maybe the text is just being butchered when its rescaled.
Though I also tried with the high detail setting which I think would deal with most issues that come from that and it didn't seem to help much