Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we used GPT-4o for image detection with 350 similar illustrations

olup-blog.pages.dev

Post reply on HN
Print view View original

41–50 of 93 posts

Re: How we used GPT-4o for image detection with 350 similar illustrations

#41

Post by babyent »

This was a fun read. I’m not a AI expert by any means. I’m also ESL. Please bear with me.

However the inaccuracy threshold seems fine for a museum, but in enterprise operations inaccuracy can mean lost revenue or worse lost trust and future business flow.

I’m struggling with some more advanced AI use cases in my collaborative work platform. I use AI (LLMs) for things like summarizations, communication, finding information using embedding. However, sometimes it is completely wrong.

To test this I spent a few days (doing something unrelated) building up a recipes database and then trying to query it for things like “I want to make a quick and easy drink”. I ran the data through classification and other steps to get as good data as I could. The results would still include fries or some other food result when I’m asking for drinks.

So I have to ask what the heck am I doing wrong? Again, for things like sending messages and reminders or coming up with descriptions, and finding old messages that match some input - no problem.

But if I have data that I’m augmenting with additional information (trying to attach more information that maybe missing but possible to deduce from what’s available) to try and enable richer workflows I’m always being bit in the butt. I feel like if I can figure this out I can provide way more value.

Not sure if what I said makes sense.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#42

Post by webmaven »

urbandw311er wrote: post #27
saint_yossarian wrote: post #9

I mean, cool tech, but why not just print a QR code next to each illustration?

This feels like one of those “NASA spent millions developing a space pen, Russians took a pencil” moments.

... which ignores the hazards of pencil shavings in a zero-g environment, especially the graphite, a good electrical conductor.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#43

Post by kjkjadksj »

Morizero wrote: post #17
Imnimo wrote: post #11

It's tough to judge without seeing examples of the targets and the user photos, but I'm curious if this could be done with just old-school SIFT. If it really is exactly the same image in the in the corpus and on the wall, does a neural embedding model really buy you a lot? A small number of high confidence tie points seems like it'd be all you need, but it probably depends a lot on just how challenging the user photo…

I find a lot of applied AI use-cases to be "same as this other method, but more expensive".

That was happening even when they were still calling it machine learning in the papers. Longer before that still. It’s the way some people reliably get papers out for better or worse. Find a known phenomenon with existing published methods, use the same dataset potentially using new method of the day, show there’s a little agreement between the old “gold standard” and your method, and boom, new paper for your cv on $hotnewmethod you can now land jobs with. Never mind no one will cite it. That’s not the point here.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#44

Post by comex »

relativ575 wrote: post #34
Imnimo wrote: post #11

It's tough to judge without seeing examples of the targets and the user photos, but I'm curious if this could be done with just old-school SIFT. If it really is exactly the same image in the in the corpus and on the wall, does a neural embedding model really buy you a lot? A small number of high confidence tie points seems like it'd be all you need, but it probably depends a lot on just how challenging the user photo…

From TFA: > LLMs and the platforms powering them are quickly becoming one-stop shops for any ML-related tasks. From my perspective, the real revolution is not the chat ability or the knowledge embedded in these models, but rather the versatility they bring in a single system. Why use another piece of software if LLM is good enough?

Performance. A museum visitor may not have a good internet connection, so any solution that involves uploading a photo to a server will probably be (much) slower than client-side detection. There’s a thin line between a magical experience and an annoying gimmick. Making people wait for something to load is a sure way to cross that line.

Also privacy. Do museum visitors know their camera data is being sent to the United States? Is that even legal (without consent) where the museum is located? Yes, visitors are supposed to be pointing their phone at a wall, but I suspect there will often be other people in view.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#45

Post by rldjbpin »

reads to me like 95% of the "conventional AI" was applied to the problem and then using llm in the end seems to work like a lucky three-faced dice.

when "embeddings" are used to perform closeness test, you are using a pretrained computer vision model behind the scenes. it is doing the far majority of tasks of filtering out hundreds of images down to a handful.

visual llm works on textual descriptions that seem far too close for similar images. regardless, more power to the team for finding something that works for them.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#46

Post by sashank_1509 »

This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than we achieved before.

My example was an image segmentation model. I managed to create an dataset of 100,000+ images and was training UNets and other advanced models on it, always reached a good validation loss but my data was simply not diverse enough and I faced a lot of issues in actual deployment, where the data distribution kept changing on a day to day basis. Then, I tried DINO v2 from Meta, finetuned on 4 images and it solved the problem, handled all the variations in lighting etc with far higher accuracy than I ever achieved. It makes sense, DINO was train on 100M + images, I would never be able to compete with that.

In this case, the company still needed my expertise, because Meta just released the weights and so someone had to setup the fine-tuning pipeline. But I can imagine a fine tuning API like OpenAI’s requiring no expertise outside of simple coding. If AI results depend on scale, it naturally follows that only a few well funded companies, will build AI that actually works, and everyone else will just use their models. The only way this trend reverses, is if compute becomes so cheap and ubiquitous, that everyone can achieve the necessary scale.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#47

Post by bboygravity »

sashank_1509 wrote: post #46

This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…

Could DINO or some other model be used to identify fillable form fields in webforms and/or PDF forms and/or desktop apps?

Or does it likely just work on real world photos and cartoons and stuff?

Re: How we used GPT-4o for image detection with 350 similar illustrations

#48

Post by pmontra »

sashank_1509 wrote: post #46

This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…

> The only way this trend reverses, is if compute becomes so cheap and ubiquitous, that everyone can achieve the necessary scale.

We would still need the 100 M+ images with accurate labels. That work can be performed collectively and open sourced but it must be maintained etc. I don't think it will be easy.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#49

Post by rolisz »

bboygravity wrote: post #47
sashank_1509 wrote: post #46

This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…

Could DINO or some other model be used to identify fillable form fields in webforms and/or PDF forms and/or desktop apps? Or does it likely just work on real world photos and cartoons and stuff?

There are dedicated models for recognizing UI elements such as form fields. One example is https://github.com/microsoft/OmniParser

Re: How we used GPT-4o for image detection with 350 similar illustrations

#50

Post by pj_mukh »

sashank_1509 wrote: post #46

This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…

Just 4 images?! Damn. I’ve had to do at least in the 100’s. I guess it depends on the complexity of the segmentation.
Post reply on HN