I mean, cool tech, but why not just print a QR code next to each illustration?
Or just a human-readable label with the model and year on it. Visitors would not need to mess with gadgets to read the labels which would be a huge usability win.
How we used GPT-4o for image detection with 350 similar illustrations
51–60 of 93 posts
Re: How we used GPT-4o for image detection with 350 similar illustrations
#52This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…
> The only way this trend reverses, is if compute becomes so cheap and ubiquitous, that everyone can achieve the necessary scale. We would still need the 100 M+ images with accurate labels. That work can be performed collectively and open sourced but it must be maintained etc. I don't think it will be easy.
DinoV2 is an unsupervised model. It learns both a high quality global image representation and local representations with no labels. It's becoming strikingly clear that foundation models are the go to choice for common data types of natural images, text, video, and audio. The labels are effectively free, the hard part now is extracting quality from massive datasets.
Re: How we used GPT-4o for image detection with 350 similar illustrations
#53This has been my experience. Foundation models have completely changed the game of ML. Previously, companies might have needed to hire ML engineers familiar with ML training, architectures etc to get mediocre results. Now companies can just hire a regular software engineer familiar with foundation model API’s to get excellent results. In some ways it is sad, but in other ways the result you get is so much better than…
This was exactly my experience being the ML engineer on a predictive maintenance project. We detected broken traffic signs in video feeds from trucks; first you segment, then you classify.
Simply yeeting every "object of interest" into DINOv2 and running any cheap classifier on that was a game changer.
Re: How we used GPT-4o for image detection with 350 similar illustrations
#54This tracks with my experience.
We built a complex processing pipeline for an NLP classification, search and comprehension task. Using vector database of Proprietary data etc.
We ran a benchmark of our system against an LLM call and the LLM performed much better for so much cheaper, in terms of dev time, complexity, and compute. Incredible time to be in working in the space seeing traditional problems eaten away by new paradigms
Re: How we used GPT-4o for image detection with 350 similar illustrations
#55I mean, cool tech, but why not just print a QR code next to each illustration?
Or just geo tag the room itself.
Re: How we used GPT-4o for image detection with 350 similar illustrations
#56I mean, cool tech, but why not just print a QR code next to each illustration?
Poster here. We would have loved that, and it was one of our first proposal - a QR code or some kind of marker. However, the client is understandably very controlling on the aesthetics of their wall as a central element of their scenography. We would have pushed for it again in the last resort, but would probably have lost the contract.
Re: How we used GPT-4o for image detection with 350 similar illustrations
#57Is there a reason to choose VGG16 over more modern models?
Yes, there is a reason. VGG16 is a lightweight model that is very cost-effective to self-host. Initially, as the article mentions, we didn’t have access to large, cost-efficient models (like AWS Titan) capable of generating image embeddings. As a result, we opted for VGG16, which is efficient, delivers good performance, and can run on CPUs with just 4GB of RAM. This makes it ideal for small-scale setups, such as VPS instances costing around €10/month.
Re: How we used GPT-4o for image detection with 350 similar illustrations
#58Calling an llm and a cv model by the same name to give the appearance of agi is a pet peeve of mine. And someone that's not openai buying into this naming convention is just unpaid propaganda
How would you prefer people talk about it? "Multimodal LLM"? My understanding is the vision portion is indeed wired directly to (and trained alongside) the language portion. > give the appearance of agi Can you point out where specifically they're doing this? Best I can tell, they give a decent summary of the effectiveness of multi-modal LLM's with support for vision, and then talk about using it to solve an incredib…
I looked into the documentation and api, and it seems you are right, it is genuinely part of the gpt model. Of course, we cannot confirm without source code.
My understanding was that there was a traditional cv library that was effectively producing an image to text before passing it to the llm. But the more I think about it, even that method would involve training for image detection to a point where objects are recognized by images not by tokens.
So the gpt product is no longer an llm or text based.
Can't say much for sure at this point with closed source, we will probably see competition catch up eventually and have more info then. At which point openai will eventually release the text2img separately and dispense with the mysticism and agi pretention.
My guess is that this is a separate image to text model ( or image+text model) and it is slapped on to the main llm code.
I don't think that text is just another modality, it probably will always be the core.
I don't have a source on something as strategic and subjective, I just have an finger on the pulse: their robot demo that does laundry, their consistent talk about AGI, their mention of power-seeking in docs, their attempt to raise trillions for chip factories, transition to for profit. They have a huge pressure to be THE monopoly and their risk is for GPT to be a text based local maximum and for intelligence not to be a sappir wolphian phenomenon.
P.s: early docs from 2023 refer to the img2txt submodel as gpt4v, that's what we should call the submodule in my opinion. (If it in fact is the same piece of tech)
Re: How we used GPT-4o for image detection with 350 similar illustrations
#59I would love to see the prompt / image data sent to GPT-4o!
On the prompt side, it's very simple, and can probably be done in a variety of ways. How we did it is to prepare a prompt with multiple "user" messages. The first one gives the instruction
you are given a reference and three candidates, which one of the candidates do you think is a match to the reference? Only output its identifier or a code when none is found
Not exactly that but something along those lines.
Then one "user" message per car (reference + candidates) with image + text indicating the type (reference or candidate) and an identifier (can be as simple as the index for the candidates).
Re: How we used GPT-4o for image detection with 350 similar illustrations
#60Side question: is there any good model that allows for image similarity detection across a large image set, that can be incrementally augmented with new images?
You'd somehow have to generate an embedding for each image, I presume.