Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we used GPT-4o for image detection with 350 similar illustrations

olup-blog.pages.dev

Post reply on HN
Print view View original

81–90 of 93 posts

Re: How we used GPT-4o for image detection with 350 similar illustrations

#81

Post by achierius »

psandor wrote: post #78
olup wrote: post #56

Earlier quoted context omitted.

Poster here. We would have loved that, and it was one of our first proposal - a QR code or some kind of marker. However, the client is understandably very controlling on the aesthetics of their wall as a central element of their scenography. We would have pushed for it again in the last resort, but would probably have lost the contract.

This is completely offtopic, but I would bet it was a government-funded museum. A reasonable institution would have worked with you to find an acceptable compromise, something much easier to implement with a small sacrifice of aesthetics. Anyway, great work, and thank you for taking the time to share it!

Really? I would much less expect a government museum to be particular about aesthetics. Privately run museums/collections/exhibitions on the other hand tend to have very finicky owners -- after all, they're putting up their own money to achieve their vision, and so of course they tend to not want to compromise on how it might look.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#82

Post by ResearchAtPlay »

jonathan-adly wrote: post #77
ResearchAtPlay wrote: post #76

Earlier quoted context omitted.

Yes, you could implement image similarity search using embeddings: Create embeddings for the entire image set, save the embeddings in a database, and add embeddings incrementally as new images come in. To search for a similar image, create the embedding for the image that you are looking for and compute the cosine similarity between that embedding and the embeddings in your database. The closer the cosine similarity…

So, there is a whole world with vision based RAG/search. We have a good open-source repo here with a ColPali implementation: https://github.com/tjmlabs/ColiVara

Thanks for the link to the ColPali implementation - interesting! I am specifically interested in evaluation benchmarks for different image embedding models.

I see the ColiVara-Eval repo in your link. If I understand correctly, ColQwen2 is the current leader followed closely by ColPali when applying those models for RAG with documents.

But how do those models compare to each other and to the llama3.2-vision embeddings when applied to, for example, sentiment analysis for photos? Do benchmarks like that exist?

Re: How we used GPT-4o for image detection with 350 similar illustrations

#83

Post by jonathan-adly »

ResearchAtPlay wrote: post #82
jonathan-adly wrote: post #77

Earlier quoted context omitted.

So, there is a whole world with vision based RAG/search. We have a good open-source repo here with a ColPali implementation: https://github.com/tjmlabs/ColiVara

Thanks for the link to the ColPali implementation - interesting! I am specifically interested in evaluation benchmarks for different image embedding models. I see the ColiVara-Eval repo in your link. If I understand correctly, ColQwen2 is the current leader followed closely by ColPali when applying those models for RAG with documents. But how do those models compare to each other and to the llama3.2-vision embeddings…

The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance.

The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t work well to pull a document where it’s a paper about birds.

1. https://arxiv.org/abs/2407.01449

Re: How we used GPT-4o for image detection with 350 similar illustrations

#84

Post by nmeofthestate »

idkman_oops wrote: post #62
olup wrote: post #2

First time for me posting this kind of story - I thought it would make an interesting case on solving a hard computer vision problem with a crafty product engineer team.

Can you tell me what font is this?

The single-character-width 'fi' ligature is quite jarring in a mono-spaced font.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#85

Post by lyu07282 »

the_duke wrote: post #60

Side question: is there any good model that allows for image similarity detection across a large image set, that can be incrementally augmented with new images? You'd somehow have to generate an embedding for each image, I presume.

Beyond using off-the-shelf embeddings, if you want to teach a model what "similar" means exactly, that's metric learning: https://paperswithcode.com/task/metric-learning

Re: How we used GPT-4o for image detection with 350 similar illustrations

#86

Post by isoprophlex »

ac2u wrote: post #67
isoprophlex wrote: post #53

Earlier quoted context omitted.

This was exactly my experience being the ML engineer on a predictive maintenance project. We detected broken traffic signs in video feeds from trucks; first you segment, then you classify. Simply yeeting every "object of interest" into DINOv2 and running any cheap classifier on that was a game changer.

Could you elaborate? I thought DINO took images and outputted segmented objects? Or do you mean that your first step was something like a yolo model to get bounding boxes and you are just using dino to segment to make the classification part easier?

We got bboxes from yolo indeed to identify "here is a traffic sign", "here is a traffic light" etc. Then we cropped out these objects of interest and took the DINOv2 embeddings of them.

Not using it to create segmentations (there are YOLO models that do that, so if you need a segmentation you can get it in one pass), no, just to get a single vector representing each crop.

Our goal was not only to know "this is a traffic sign", but also do multilabel classification like "has graffiti", "has deformations", "shows decoloration" etc. If you store those it becomes pretty trivial (and hella fast) to pass these off to a bunch of data scientists so they can let loose all the classifiers in sklearn on that. See [1] for a substantially similar example.

[1] https://blog.roboflow.com/how-to-classify-images-with-dinov2

Re: How we used GPT-4o for image detection with 350 similar illustrations

#87

Post by ResearchAtPlay »

jonathan-adly wrote: post #83
ResearchAtPlay wrote: post #82

Earlier quoted context omitted.

Thanks for the link to the ColPali implementation - interesting! I am specifically interested in evaluation benchmarks for different image embedding models. I see the ColiVara-Eval repo in your link. If I understand correctly, ColQwen2 is the current leader followed closely by ColPali when applying those models for RAG with documents. But how do those models compare to each other and to the llama3.2-vision embeddings…

The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance. The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t wo…

Makes sense. My main takeaway from the ColPali paper (and your comments) is that ColPali works best for document RAG, whereas vision model embeddings are best used for image similarity search or sentiment analysis. So to answer my own question: The best model to use depends on the application.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#88

Post by hackerdood »

Very neat explanation of solving these kinds of unique challenges, especially given how similar the illustrations were.

One question I had was, knowing how difficult it was to train the model with the base images, and given that the client didn’t have time to photograph them, did you consider flying someone out to the museum for a couple of days to photograph each illustration from several angles with the actual lighting throughout the day? Or potentially hiring a photographer near the museum to do that? It seems like a round trip ticket plus a couple nights in a hotel could have saved a lot of headache, providing more images to turn into synthetic training data. Even if you still had to resort to using 4o as a tiebreaker, it could be that you only present two candidates as the third might have a much lower similarity score to the second candidate. Good write up either way.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#89

Post by ac2u »

isoprophlex wrote: post #86
ac2u wrote: post #67

Earlier quoted context omitted.

Could you elaborate? I thought DINO took images and outputted segmented objects? Or do you mean that your first step was something like a yolo model to get bounding boxes and you are just using dino to segment to make the classification part easier?

We got bboxes from yolo indeed to identify "here is a traffic sign", "here is a traffic light" etc. Then we cropped out these objects of interest and took the DINOv2 embeddings of them. Not using it to create segmentations (there are YOLO models that do that, so if you need a segmentation you can get it in one pass), no, just to get a single vector representing each crop. Our goal was not only to know "this is a traf…

Understood. Thanks for taking the time to elaborate.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#90

Post by numba888 »

babyent wrote: post #41

This was a fun read. I’m not a AI expert by any means. I’m also ESL. Please bear with me. However the inaccuracy threshold seems fine for a museum, but in enterprise operations inaccuracy can mean lost revenue or worse lost trust and future business flow. I’m struggling with some more advanced AI use cases in my collaborative work platform. I use AI (LLMs) for things like summarizations, communication, finding inform…

> Not sure if what I said makes sense.

Not sure either. But here is the lesson from this and other sources. To improve the output use multistep approach. Get the first answer, one or more, and pass it through the second verification step(s). Like 'for this * is this *' relevant? Or is it correct, does it solve the problem, etc.. Then select the answer with the best scores on the filters. You see, it's very similar to that in the original post. Get first candidates, filter.

Post reply on HN