Earlier quoted context omitted.
Would love to see the test file.
would be glad to see benchmarking results
This is a good idea. We should publish a benchmark results/comparison.
Mistral OCR
111–120 of 450 posts
Re: Mistral OCR
#112Co-founder of doctly.ai here (OCR tool) I love mistral and what they do. I got really excited about this, but a little disappointed after my first few tests. I tried a complex table that we use as a first test of any new model, and Mistral OCR decided the entire table should just be extracted as an 'image' and returned this markdown: ```  ``` I'll keep testing, but so far, very disappointing…
If you have a judge system, and Mistral performs well on other tests, wouldn't you want to include it so if it scores the highest by your judges ranking it would select the most accurate result? Or are you saying that mistral's image markdown would score higher on your judge score?
Re: Mistral OCR
#113Looks good but in the first hover/slider demo one can see how it could lead to confusion when handling side by side content.
Table 1 is referred to in section `2 Architectural details` but before `2.1 Multimodal Decoder`. In the generated markdown though it is below the latter section, as if it was in/part of that section.
Of course I am nitpicking here but just the first thing I noticed.
Re: Mistral OCR
#114This is cool! With that said for anyone looking to use this in RAG, the downside to specialized models instead of general VLMs is you can't easily tune it to your use specific case. So for example, we use Gemini to add very specific alt text to images in the extracted Markdown. It's also 2 - 3X the cost of Gemini Flash - hopefully the increased performance is significant.
Regardless excited to see more and more competition in the space.
Wrote an article on it: https://www.sergey.fyi/articles/gemini-flash-2-tips
Re: Mistral OCR
#115Earlier quoted context omitted.
We’ll just stick LLM Gateway LLM in front of all the specialized LLMs. MicroLLMs Architecture.
I actually think you're onto something there. The "MicroLLMs Architecture" could mirror how microservices revolutionized web architecture. Instead of one massive model trying to do everything, you'd have specialized models for OCR, code generation, image understanding, etc. Then a "router LLM" would direct queries to the appropriate specialized model and synthesize responses. The efficiency gains could be substantial…
I’m doing this personally for my own project - essentially building an agent graph that starts with the image output, orients and cleans, does a first pass with tesseract LSTM best models to create PDF/HOCR/Alto, then pass to other LLMs and models based on their strengths to further refine towards markdown and latex. My goal is less about RAG database population but about preserving in a non manually typeset form the structure and data and analysis, and there seems to be pretty limited tooling out there since the goal generally seems to be the obviously immediately commercial goal of producing RAG amenable forms that defer the “heavy” side of chart / graphic / tabular reproduction to a future time.
Re: Mistral OCR
#116Earlier quoted context omitted.
Not OP, but it sounds like the kind of project I’d undertake. Happiness for me is about exploring the problem within constraints and the satisfaction of building the solution. Brittleness is often of less concern than the fun factor. And some kinds of brittleness can be managed/solved, which adds to the fun.
I would posit that learning how the device works, and how to integrate with a newer digital monitoring device would be just as interesting and less brittle.
Possibly! But I’ve recently wanted to dabble with computer vision, so I’d be looking at a project like this as a way to scratch a specific itch. Again, not OP so I don’t know what their priorities are, but just offering one angle for why one might choose a less “optimal” approach.
Re: Mistral OCR
#117Co-founder of doctly.ai here (OCR tool) I love mistral and what they do. I got really excited about this, but a little disappointed after my first few tests. I tried a complex table that we use as a first test of any new model, and Mistral OCR decided the entire table should just be extracted as an 'image' and returned this markdown: ```  ``` I'll keep testing, but so far, very disappointing…
Interestingly I’m currently going through and scanning the hundreds of journal papers my grandfather authored in medicine and thinking through what to do about graphs. I was expecting to do some form of multiphase agent based generation of LaTeX or SVG rather than a verbal summary of the graphs. At least in his generation of authorship his papers clearly explained the graphs already. I was pretty excited to see your…
We need to update the examples on the front page. Currently for things that are considered charts/graphs/figures we convert to a description. For things like logos or images we do an image tag. You can also choose to exclude them.
The difference with this is that it took the entire page as an image tag (it's just a table of text in my document). rather than being more selective.
I do like that they give you coordinates for the images though, we need to do something like that.
Give the actual tool a try. Would love to get your feedback for that use case. It gives you 100 free credits initially but if you email me (ali@doctly.ai), I can give you an extra 500 (goes for anyone else here also)
Re: Mistral OCR
#118Dupe of an hour previous post
https://news.ycombinator.com/item?id=43282489
Re: Mistral OCR
#119Wonder how it does with table data in pdfs / page-long tabular data?
Re: Mistral OCR
#120Dang. Super fast and significantly more accurate than google, Claude and others. Pricing : $1/1000 pages, or per 2k pages if “batched”. I’m not sure what batching means in this case: multiple pdfs? Why not split them to halve the cost? Anyway this looks great at pdf to markdown.
From my testing so far, it seems it's super fast and responded synchronously. But it decided that the entire page is an image and returned `` with coordinates in the metadata for the image, which is the entire page. Our tool, doctly.ai is much slower and async, but much more accurate and gets you the content itself as an markdown.
I thought we stopped -ly company names ~8 years ago?