Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

First Impressions with GPT-4V(ision)

blog.roboflow.com

Post reply on HN
Print view View original

291–300 of 345 posts

Re: First Impressions with GPT-4V(ision)

#291

Post by gdubs »

kristopolous wrote: post #151

This actually doesn't seem like it's a giant lift using modern image classifiers. The basic idea is to use diffusion classifiers to caption the image to generate descriptive text and append the prompt. The work part is getting the ensemble right since you'll need to use a general classifier, like BLIP, to identify say a bunch of text from a plant and then, in this example, use structured OCR and pl@ntnet to get more…

The main difference here is that GPT4 is truly multimodal, and so the model itself understands images and doesn't require a preprocessor to tell it what's in the image. As you show, that's a viable approach – but it has limitations. Mainly, things get lost in translation.

It's like the difference between you telling me what's outside the window and then asking me questions about it – versus me being able to look out the window myself.

Re: First Impressions with GPT-4V(ision)

#292

Post by giancarlostoro »

cs702 wrote: post #10

Sure, there are a few edge-case failures and mistakes here and there, but I can't help but be in awe . AWE. Let me state the obvious, in case anyone here isn't clear about the implications: If the rate of improvement of these AI models continues at the current pace, they will become a superior user interface to almost every thing you want to do on your mobile phone, your tablet, your desktop computer, your car, your…

I think at this point the cost factor needs to be near zero for this to explode the way we want. People think I'm "moving the goal post" when I say you literally will see it explode once it all runs off a cell phone, offline, without internet. Because nobody will want to pay money to use an AI UI when the non-AI UI was free.

Re: First Impressions with GPT-4V(ision)

#294

Post by SoftTalker »

chx wrote: post #283
SoftTalker wrote: post #274

Earlier quoted context omitted.

I often fail at those too. If a square contains a small corner of a thing, but is mostly not the thing, do I check it or not?

read the problem again please. it's not that it misidentified what's in the squares, it "miscounted" the number of squares -- because it doesn't count, it's just a plausible sentence generator.

Proving that CAPTCHAs are a poor test for "are you human" as we both failed.

Re: First Impressions with GPT-4V(ision)

#296

Post by a_wild_dandan »

stri8ted wrote: post #263
blovescoffee wrote: post #260

Earlier quoted context omitted.

How what works? Could you elaborate?

As far as I understand, these multi-modal models work by embedding the text/image in a shared representation space. To perform OCR on such an embedding, it would require extracting every letter, in the correct order, from the embedding. But given the embedding is a fixed size, and therefor necessarily compressed, I would expect it to loose the exactness of the underlying input, especially with images containing a lot…

You're correct! Feature extractors lose fidelity and have finite attention, just like us. But we can reduce/compress the "essence" of an image, paragraph, song, etc into some combination of underlying features.

Think of a 4096x4096 pixel white image.

To hold this image in mind, does your memory load tens of millions of bits? Thankfully no! What if we add a big red circle which spans the image? Or write the chorus of All Star inside it? Ezpz! The number of "features" is comically simple.

Same thing for AI models. They discover the concept of letters, the sound of b-flats, image symmetry, turns of phrase, the conceptual distance between a "woman" an a "queen", etc. These are all natural patterns common to the data it sees. It can thus (like us!) reduce complicated input into a (fixed-size) smear of these learned, related features.

Re: First Impressions with GPT-4V(ision)

#297

Post by famouswaffles »

ldhough wrote: post #207

Oddly just like the text version it is still really bad at tic-tac-toe. Gave it a picture of a completed game and "Who won?" It told me "X won with a vertical line through the middle column" when in fact O won and there was only one X in the middle column. Very impressive with almost everything else I gave it though.

https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47...

You can get optimal tic tac toe with painstaking instructions

Re: First Impressions with GPT-4V(ision)

#298

Post by Jackson__ »

idontknoworcare wrote: post #247
famouswaffles wrote: post #11

Graph analysis is impressive (last example) - https://imgur.com/a/iOYTmt0 Can do UI to frontend. Seems to understand the UI graphical elements and layout, not just text https://twitter.com/skirano/status/1706823089487491469 Can describe comic images accurately, panel by panel - https://twitter.com/ComicSociety/status/1698694653845848544?... Lots of examples here also - https://www.reddit.com/r/ChatGPT/comments/16sdac…

In your twitter's comic book link the first image has a women with a huge ass with no pants on. The ass is colored grey and so the lack of clothes is not noticeable until you click into the image. How many children here on hacker news are going to see this and get addicted to porn? Perhaps a few. You deserve to be banned.

>How many children here on hacker news are going to see this and get addicted to porn?

Billions. I propose we shut down hacker news entirely, as it seems unfathomable the good will ever be able to outweigh the bad at this rate.

Re: First Impressions with GPT-4V(ision)

#299

Post by ldhough »

famouswaffles wrote: post #297
ldhough wrote: post #207

Oddly just like the text version it is still really bad at tic-tac-toe. Gave it a picture of a completed game and "Who won?" It told me "X won with a vertical line through the middle column" when in fact O won and there was only one X in the middle column. Very impressive with almost everything else I gave it though.

https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47... You can get optimal tic tac toe with painstaking instructions

That is pretty interesting and also I didn't realize you could share chats like that.

That GPT is so bad at tic-tac-toe and relatively good at other games like chess is one of the main things that contributes to me having a lower opinion of its ability to generalize than I would have otherwise.

I think any human with GPT's abilities in chess (but somehow no prior knowledge of ttt) would have zero issue becoming an expert with a single explanation of the game. Even very young children can learn to play ttt well and at least consistently make valid moves if nothing else.

Re: First Impressions with GPT-4V(ision)

#300

Post by wilg »

dumpsterdiver wrote: post #290
wilg wrote: post #264

Earlier quoted context omitted.

There’s no reason chatbots have to be the interface to an LLM. Imagine dynamically generated interfaces redesigning themselves to your needs as you work through a task.

So to your point, I do think it would be nice to have an AI assistant suggest to make changes to the view mode of an application. For instance, if I navigate to a certain area the AI might suggest that I make that option visible in my menu. I might say no, because I prefer to use the hotkey. Aside from that, I'm going to have to absolutely side with the parent commenter on this one - When I'm doing serious work I wan…

I think there’s room for an AI-native interface that knows the full context of the project you’re working on and can make extremely intelligent suggestions or tailor the interface for a specific task. Or you could just literally describe the interface you want for something and it redesigns it for you immediately.
Post reply on HN