Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OCR Benchmark Focusing on Automation

nanonets.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Show HN: OCR Benchmark Focusing on Automation

#11

Post by kapitalx »

prats226 wrote: post #10
kapitalx wrote: post #7

Great list! I’ll definitely run your benchmark against Doctly.ai (our PDF-to-Markdown service) specially as we publish our workflow service, to see how we stack up. One thing I’ve noticed in many benchmarks, though, is the potential for bias. I’m actually working on a post about this issue, so it’s top of mind for me. For example, in the omni benchmark, the ground truth expected a specific order for heading informati…

Bias wrt ordering is a great point. What we consider structured information in this benchmark is irrespective of how its presentation (Order, format etc), it should be directly comparable. So the benchmark does that it into account. Example is if you are only converting lets say an invoice into markdown, you can introduce bias wrt ordering etc. But if the task is to find out invoice number, total amount, number of li…

Exactly. You still have to be explicit in order to remove bias. Either by sorting the keys, or looking up specific keys. For arrays, I would say order still matters. For example when you capture a list of invoice items, you should maintain order.

Re: Show HN: OCR Benchmark Focusing on Automation

#13

Post by themanmaran »

kapitalx wrote: post #7

Great list! I’ll definitely run your benchmark against Doctly.ai (our PDF-to-Markdown service) specially as we publish our workflow service, to see how we stack up. One thing I’ve noticed in many benchmarks, though, is the potential for bias. I’m actually working on a post about this issue, so it’s top of mind for me. For example, in the omni benchmark, the ground truth expected a specific order for heading informati…

Hey I wrote the Omni benchmark. I think you might be misreading the methodology on our side. Order on page does not matter in our accuracy scoring. In fact we are only scoring on JSON extraction as a measurement of accuracy. Which is order independent.

We chose this method for all the same reasons you highlight. Text similarity based measurements are very subject to bias, and don't correlate super well with accuracy. I covered the same concepts in the "The case against text-similarity"[1] section of our writeup.

[1] https://getomni.ai/ocr-benchmark

Re: Show HN: OCR Benchmark Focusing on Automation

#15

Post by derrasterpunkt »

sumedh wrote: post #8
criddell wrote: post #5

How do these compare to traditional commercial and open source OCR tools? What about things like the Apple Vision APIs?

> Apple Vision APIs Can we even OCR pdfs using Apple Vision APIs

You can[1].

I’m vibe coding a little macOS OCR app since last weekend, and I’m really happy with the results so far. This is my first app, so fingers crossed. If it becomes feature-complete and polished enough, I’m considering open sourcing it. There’s still a long way to go, though.

[1] https://developer.apple.com/documentation/vision/vnrecognize...

Re: Show HN: OCR Benchmark Focusing on Automation

#16

Post by kapitalx »

themanmaran wrote: post #13
kapitalx wrote: post #7

Great list! I’ll definitely run your benchmark against Doctly.ai (our PDF-to-Markdown service) specially as we publish our workflow service, to see how we stack up. One thing I’ve noticed in many benchmarks, though, is the potential for bias. I’m actually working on a post about this issue, so it’s top of mind for me. For example, in the omni benchmark, the ground truth expected a specific order for heading informati…

Hey I wrote the Omni benchmark. I think you might be misreading the methodology on our side. Order on page does not matter in our accuracy scoring. In fact we are only scoring on JSON extraction as a measurement of accuracy. Which is order independent. We chose this method for all the same reasons you highlight. Text similarity based measurements are very subject to bias, and don't correlate super well with accuracy.…

I'll dig deeper into your code, but scanning your post does look like your are addressing this. That's great.

If I do find anything, I'll share with you for comments before I publish the post.

Re: Show HN: OCR Benchmark Focusing on Automation

#17

Post by 29athrowaway »

Many of the benchmarks I have seen in this space suffer from the Texas Sharpshooter fallacy, where you shoot first and then paint a target around the hole.

If you create a benchmark and your product outperforms everything else, it could mean many things. Overfitting being one of them.

Re: Show HN: OCR Benchmark Focusing on Automation

#18

Post by prats226 »

29athrowaway wrote: post #17

Many of the benchmarks I have seen in this space suffer from the Texas Sharpshooter fallacy, where you shoot first and then paint a target around the hole. If you create a benchmark and your product outperforms everything else, it could mean many things. Overfitting being one of them.

That's an interesting point. The bias might or might not be intentional. From the benchmarks I have seen, lot of tools solve slightly different problems altogether and also target slightly different data distribution and in the end have to build best solution around it.

Which is why publishing open benchmarks is first step where there is public scrutiny around whether then benchmark itself irrespective of the results is fair. In the end, the end user will choose the benchmark that's best fit for their usecase or mostly will create a variation of their own, do their own unbiased evaluations.

Re: Show HN: OCR Benchmark Focusing on Automation

#19

Post by pronoiac »

sumedh wrote: post #8
criddell wrote: post #5

How do these compare to traditional commercial and open source OCR tools? What about things like the Apple Vision APIs?

> Apple Vision APIs Can we even OCR pdfs using Apple Vision APIs

I've used ocrit, which uses those APIs. https://github.com/insidegui/ocrit

There are also:

* swiftocr - https://github.com/fny/swiftocr

* macos-vision-ocr - https://github.com/bytefer/macos-vision-ocr

Re: Show HN: OCR Benchmark Focusing on Automation

#20

Post by serjester »

prats226 wrote: post #14
arresin wrote: post #12

In your pricing example I see $6.27 for a 10 page document. That is extremely expensive.

Can share link? Maybe some kind of mistake.

You guys have some of the most complex pricing I’ve ever seen. I tried to read it a couple times and I still have no idea what any of this costs.
Post reply on HN