Hasn't been done (well) yet, and it's absolutely doable. BUT - I don't think that's actually one of the big problems to be solved. I would argue there are enough knowledgable eyeballs out there that there really aren't too too many "undiscovered" rarities.
And what I mean by that is, for example: The 1909-S VDB or the 1955 Doubled Die are two of the mega rarities when it comes to Lincoln cents. They are worth $500+ even in the worst shape. But they're very obvious to someone with even basic coin knowledge, and there are lots of books and pictures out there. And in fact, I'd argue there are very very very few of those coins out there that haven't been identified for what they are -- they've been known rarities for 114 and 68 years respectively. So computer vision telling someone that it's a rare coin isn't going to be too much of a game-changer.
(This is a strongly stated, loosely held belief btw - I can easily be argued into a different perspective).
I think a much more interesting application for computer vision to the business is when it comes to grading (coins are graded on a 1-70 scale for Reasons). And in some cases, the difference a single grade point makes can be $thousands. Official grading is still done entirely by human eyes. An efficient computer vision system to look for undergraded coins I think has real opportunity. It's not an easy feat though - you can't really truly grade with static photos, and there needs to be normalization, so you'd need video clips w/ depth mapping of many thosands of coins to train a model properly. But it could be done!