Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama-OCR: Document to Markdown

llamaocr.com

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Llama-OCR: Document to Markdown

#51

Post by gunzel412 »

notsylver wrote: post #26
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. For Normal models, the state of Open Source OCR is pretty terrible. Unfortunately, the closed options from Microsoft, Google etc are much better. Did you try those ? Interesting about Flash, what LLMs did you test ?

I tried open source and closed source OCR models, all were pretty bad. Google vision was probably the best of the "OCR" models, but it liked adding spaces between characters and had other issues I've forgotten. It was bad enough that I wondered if I was using it wrong. By the time I was trying to pass the text to an LLM with the image so it could do "touchups" and fix the mistakes, I gave up and decided to try LLMs f…

[dead]

Re: Llama-OCR: Document to Markdown

#52

Post by Eisenstein »

Curiositry wrote: post #41
nutlope wrote: post #22

Hi all, I'm the author of llama-ocr. Thank you for sharing & for the kind comments! I built this earlier this week since I wanted a simple API to do OCR – it uses llama 3.2 vision (hosted on together.ai, where i work) to parse images into structured markdown. I also have it available as an npm package. Planning to add a bunch of other features like the ability to parse PDFs, output a response in JSON, ect... If anyon…

Option to use a local LLM?

I made a script which does exactly the same thing but locally using koboldcpp for inference. It downloads MiniCPM-V 2.6 with image projector the first time you run it. If you want to use a different model you can, but you will want to edit the instruct template to match.

* https://github.com/jabberjabberjabber/LLMOCR

Re: Llama-OCR: Document to Markdown

#53

Post by mosselman »

philips wrote: post #27

I have recently used llama3.2-vision to handle some paper bidsheets for a charity auction and it is fairly accurate with some terrible handwriting. I hope to use it for my event next year. I do find it rather annoying not being able to get it to consistently output a CSV though. ChatGPT and Gemini seem better at doing that but I haven’t tried to automate it. The scale of my problem is about 100 pages of bidsheets and…

What about using llama3.2-vision to do the OCR bit and then deferring to ChatGPT to do the CSV part?

Re: Llama-OCR: Document to Markdown

#54

Post by mg »

Vetch wrote: post #47
mg wrote: post #34

I gave it a sentence, which I created by placing 500 circles via a genetic algorithm to form a sentence. And then drew with an actual physical circle: https://www.instagram.com/marekgibney/p/BiFNyYBhvGr/ Interestingly, it sees the circles just fine, but not the sentence. It replied with this: The image contains no text or other elements that can be represented in Markdown. It is a visual composition of circles and do…

Based on the fact that squinting works, I applied a Gaussian blur to the image. Here's the response I got: Markdown: The provided image is a blurred text that reads "STOP THINKING IN CIRCLES." There are no other visible elements such as headers, footers, subtexts, images, or tables. Markdown Content: STOP THINKING IN CIRCLES As the response is not deterministic, I also tried several times with the unprocessed image b…

I guess blurring it is similar to reducing the resolution or to looking at the image from further away.

It's interesting that the neural net figures out the circles, but not the words. Because the circles are also not so easily apparent from looking closely at the image. It could also be whirly lines.

Re: Llama-OCR: Document to Markdown

#55

Post by nirav72 »

Eisenstein wrote: post #52
Curiositry wrote: post #41

Earlier quoted context omitted.

Option to use a local LLM?

I made a script which does exactly the same thing but locally using koboldcpp for inference. It downloads MiniCPM-V 2.6 with image projector the first time you run it. If you want to use a different model you can, but you will want to edit the instruct template to match. * https://github.com/jabberjabberjabber/LLMOCR

MiniCPM-v 2.6 is probably the best self-hosted vision model I have used so far. Not just for OCR, but also image analysis. I have it setup, so my NVR (frigate) sends couple of images upon motion alert from a driveway security camera to Ollama with minicpm-v 2.6. I’m able to get a reasonably accurate description of the vehicle that pulled into the driveway. Including describing the person that exits the vehicle and also the license plate. All sent to my phone.

Re: Llama-OCR: Document to Markdown

#59

Post by dleeftink »

notsylver wrote: post #26
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Normal OCR models are terrible at it, LLMs do far better. Gemini Flash came out on top from the models I tested and it wasn't even close. For Normal models, the state of Open Source OCR is pretty terrible. Unfortunately, the closed options from Microsoft, Google etc are much better. Did you try those ? Interesting about Flash, what LLMs did you test ?

I tried open source and closed source OCR models, all were pretty bad. Google vision was probably the best of the "OCR" models, but it liked adding spaces between characters and had other issues I've forgotten. It was bad enough that I wondered if I was using it wrong. By the time I was trying to pass the text to an LLM with the image so it could do "touchups" and fix the mistakes, I gave up and decided to try LLMs f…

WordNinja is pretty good as a post-processing step on wrongly split/concatenated words:

[0]: https://github.com/keredson/wordninja

Re: Llama-OCR: Document to Markdown

#60

Post by criddell »

alecco wrote: post #57

Is it possible to do this locally with open source software? I have a lot of accounting PDFs to convert but due to privacy concerns it should not run in the cloud.

Does it have to be open source, or just running locally? The paid version of Acrobat does this well. MacOS has pretty good built-in OCR capabilities and Windows isn’t far behind.

If you have the hardware for it, you can run some LLMs locally. Although for accounting data, I probably wouldn’t trust it.

Post reply on HN