Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we used GPT-4o for image detection with 350 similar illustrations

olup-blog.pages.dev

Post reply on HN
Print view View original

31–40 of 93 posts

Re: How we used GPT-4o for image detection with 350 similar illustrations

#31

Post by JayShower »

Alternative solution that would require less heavy lifting of ML but a little more upfront programming: It sounds like the cars are arranged in a grid on the wall. Maybe it would be possible to narrow down which car the user took a photo of by looking at the photos of the surrounding cars as well, and hardcoding into the system the position of each car relative to one another? Could potentially do that locally very quickly (maybe even at the level of QR-code speed) versus doing an embedding + LLM.

Con of this approach would be that it’s requires maintenance if they ever decide to change the illustration positions.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#32

Post by yannis »

olup wrote: post #2

First time for me posting this kind of story - I thought it would make an interesting case on solving a hard computer vision problem with a crafty product engineer team.

Thanks for sharing. Interesting approach. As other commenters mentioned, article could do well with some hypothetical images. Maybe on a follow-up blog post? Also since you mentioning your Company's name you missing an opportunity for marketing by not providing a link.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#33

Post by wongarsu »

brody_hamer wrote: post #24
wongarsu wrote: post #22

Interesting approach to a a very interesting challenge, given how close the images supposedly are. With the limited training data they have I'm surprised they don't mention any attempts at synthetic training data. Make (or buy) a couple museum scenes in blender, hang one of the images there, take images from a lot of angles, repeat for more scenes, lighting conditions and all 350 images. Should be easy to script. The…

They did. > “ To address this limitation, we turned to data augmentation, artificially creating new versions of each image by modifying colors, adding noise, applying distortion, or rotating images. By the end, we had generated 600 augmented images per car.”

Those are pretty standard. A standard YOLO training run applies more transformations than that, and there are ready-made modules that do the same in keras and pytorch (for their mobilenet and VGG16). I'm not sure if anyone is training any serious vision algorithm without that kind of data augmentation.

What I am talking about is that they want to recognize scenes containing the images, but only have the images as training data. They have a good idea what those scenes will look like. Going there to take actual training pictures was evidently not viable, but generating approximations of them might have been.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#34

Post by relativ575 »

Imnimo wrote: post #11

It's tough to judge without seeing examples of the targets and the user photos, but I'm curious if this could be done with just old-school SIFT. If it really is exactly the same image in the in the corpus and on the wall, does a neural embedding model really buy you a lot? A small number of high confidence tie points seems like it'd be all you need, but it probably depends a lot on just how challenging the user photo…

From TFA:

> LLMs and the platforms powering them are quickly becoming one-stop shops for any ML-related tasks. From my perspective, the real revolution is not the chat ability or the knowledge embedded in these models, but rather the versatility they bring in a single system.

Why use another piece of software if LLM is good enough?

Re: How we used GPT-4o for image detection with 350 similar illustrations

#35

Post by relativ575 »

Morizero wrote: post #17
Imnimo wrote: post #11

It's tough to judge without seeing examples of the targets and the user photos, but I'm curious if this could be done with just old-school SIFT. If it really is exactly the same image in the in the corpus and on the wall, does a neural embedding model really buy you a lot? A small number of high confidence tie points seems like it'd be all you need, but it probably depends a lot on just how challenging the user photo…

I find a lot of applied AI use-cases to be "same as this other method, but more expensive".

Use cases such as?

Re: How we used GPT-4o for image detection with 350 similar illustrations

#36

Post by titzer »

relativ575 wrote: post #34
Imnimo wrote: post #11

It's tough to judge without seeing examples of the targets and the user photos, but I'm curious if this could be done with just old-school SIFT. If it really is exactly the same image in the in the corpus and on the wall, does a neural embedding model really buy you a lot? A small number of high confidence tie points seems like it'd be all you need, but it probably depends a lot on just how challenging the user photo…

From TFA: > LLMs and the platforms powering them are quickly becoming one-stop shops for any ML-related tasks. From my perspective, the real revolution is not the chat ability or the knowledge embedded in these models, but rather the versatility they bring in a single system. Why use another piece of software if LLM is good enough?

Cost. Same reason you don't deliver UPS packages with B-2 bombers.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#37

Post by throwaway314155 »

TZubiri wrote: post #28

Calling an llm and a cv model by the same name to give the appearance of agi is a pet peeve of mine. And someone that's not openai buying into this naming convention is just unpaid propaganda

How would you prefer people talk about it? "Multimodal LLM"? My understanding is the vision portion is indeed wired directly to (and trained alongside) the language portion.

> give the appearance of agi

Can you point out where specifically they're doing this? Best I can tell, they give a decent summary of the effectiveness of multi-modal LLM's with support for vision, and then talk about using it to solve an incredibly narrow task. The only diction I could see that hints at "agi" is when they describe the versatility of this approach; but how could you possibly argue against that? It's objectively more versatile (if not wasteful and more expensive).

Re: How we used GPT-4o for image detection with 350 similar illustrations

#38

Post by Morizero »

relativ575 wrote: post #35
Morizero wrote: post #17

Earlier quoted context omitted.

I find a lot of applied AI use-cases to be "same as this other method, but more expensive".

Use cases such as?

I'm in an AI focused education research group, and most "smart/personalized tutors" on the market have similar processes and outcomes as paper flashcards.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#39

Post by msp26 »

titzer wrote: post #36
relativ575 wrote: post #34

Earlier quoted context omitted.

From TFA: > LLMs and the platforms powering them are quickly becoming one-stop shops for any ML-related tasks. From my perspective, the real revolution is not the chat ability or the knowledge embedded in these models, but rather the versatility they bring in a single system. Why use another piece of software if LLM is good enough?

Cost. Same reason you don't deliver UPS packages with B-2 bombers.

The cost of LLM inference is cheap and will continue to decrease. More traditional methods take up far more of an engineer's time (which also costs money).

If I have a project with a low enough lifetime inputs I'm not wasting my time labelling data and training a model. That time could be better spent working on something else. As long as the evaluation is thorough, it doesn't matter. But I still like doing some labelling manually to get a feel for the problem space.

Re: How we used GPT-4o for image detection with 350 similar illustrations

#40

Post by Terr_ »

mattnewton wrote: post #29
Terr_ wrote: post #19

Earlier quoted context omitted.

Better to spend $100 in op-ex money than spend $1 in cap-ex money reading a journal paper, especially if it lets you tell investors "AI." :p

Your engineers cost <$1/hr and understand journal papers?

The 100-vs-1 is a ratio.
Post reply on HN