Live data from Hacker News

First Impressions with GPT-4V(ision)

blog.roboflow.com

71–80 of 345 posts

Re: First Impressions with GPT-4V(ision)

#71
post #10

Sure, there are a few edge-case failures and mistakes here and there, but I can't help but be in awe . AWE. Let me state the obvious, in case anyone here isn't clear about the implications: If the rate of improvement of these AI models continues at the current pace, they will become a superior user interface to almost every thing you want to do on your mobile phone, your tablet, your desktop computer, your car, your…

I agree. I think apps that would initially benefit from LLM-powered conversational interfaces are those that have the following traits:

  - constrained context
  - part of a hands-free workflow
A couple use-cases I have been pondering are driving assistant and cooking assistant.

People are already used to using their phone or car's nav system to give them directions to an unfamiliar place. But even with such a system it's useful to have a human navigator in the car with you to answer various questions:

  - What's my next turn again?
  - How long till we get there?
  - Are there any rest stops near here?
  - What was that restaurant we just passed?
  - Is there another route with less traffic?
These questions are all answerable with context that can be provided by the mapping app:

  - List of upcoming directions
  - Overall route planning
  - Surrounding place data
  - Traffic data and alternate route information
It's possible to pull over to the side of the road, take off your distance glasses, put on your reading glasses, and zoom/pan the map to try to answer these questions yourself. But if the map application can just expose its API to the language interface layer, then a user can get the answers without taking their eyes off the road.

The information is contextual and constrained based on a current task. In some cases it might be more desirable to whip out your phone and interact with the map to look up the answers on a screen, but often it won't be worth stopping the car, and so the conversational interface is better.

Cooking assistant is a similar case: you are busy stirring something and checking on the oven -- you don't want to wipe the flour off your hands to pick up your phone and ask how many teaspoons of sugar you need. Again: contextual and constrained info based on a current task, and your hands and eyes -- the instruments of traditional UIs -- are otherwise occupied.

Today, our software interfaces generally have one of two kinds of entity on the other end: humans, or other software. In the near future there will be another type of entity: language models. We need to start thinking of how our APIs will change when they're interacting with an LLM -- e.g. they'll need to be discoverable and self-describing; error states will need to be standardized or explicit with instructions on how to correct; they'll need to be fast enough to fit in a conversational interface; etc. It's arguable that such traits are part of good API design today, but in the future they may be required for the API to function in a landscape of virtual agents.

Re: First Impressions with GPT-4V(ision)

#72
post #23

Earlier quoted context omitted.

Your job will change in fundamental ways at least.

The more people say that, the less convincing it is There is no way I would have a UI developer onboarded when I can generate many iterations of layouts in midjourney, copy them into chatgpt4 and get code in NextJS with Typescript instantly non devs will have trouble doing this or thinking of the prompts to ask, but the dev team asking for headcount simply wont ask for headcount, and the engineering manager is going…

Yeah, I'm also skeptical about the actual value of specialists in the future. To me, AI brings a ton of power to generalists, who now have access to very powerful tools that would have taken them a long time to learn otherwise.

Re: First Impressions with GPT-4V(ision)

#73
post #28
post #10

Sure, there are a few edge-case failures and mistakes here and there, but I can't help but be in awe . AWE. Let me state the obvious, in case anyone here isn't clear about the implications: If the rate of improvement of these AI models continues at the current pace, they will become a superior user interface to almost every thing you want to do on your mobile phone, your tablet, your desktop computer, your car, your…

I don’t know, I hate the idea of having to hold a natural-language conversation with a computer in order to make use of its functionality. It feels like being one of those Futurama heads in a jar that can’t do anything by themselves.

UIs being dumbed down for average users was already annoying. Apparently the process won't stop until the illiterate are included too.

Re: First Impressions with GPT-4V(ision)

#75

Earlier quoted context omitted.

Did you read my comment? I literally said that it’s for vision impaired. That’s great. …but it’s niche . I’m sitting on my couch right now and I can think of like 20 things I could chat to chatgpt about. I can see literally nothing in my visual range want to take a photo of and run image analysis over. It’s like Shazam. Yes, it’s useful, but, most of the time, I don’t need it. I would argue this is true for this, for…

There's enough vision-impaired people in the world to equal the population of Japan, Korea, and Vietnam combined. And beyond those people who would get obvious utility, this is essentially Google Lens on steroids—I simply can't figure how you could call this "niche". Maybe you won't use it multiple times per day, but plenty of people will. Hell, just now I was wondering why the leaves on one of my plants are starting…

Many of those people would be adequately served with a pair of glasses.

This, wouldn't help them, even if they had both a device capable of using it and the means to pay for it.

Even if it could help people, it's an open question if it would be safe, to, for example, use this to scan medication when it is only a probabilistic model that may hallucinate something that isn't actually there.

What you're talking about is a speculative use of a service that might one day exist based on this technology.

What I am talking about is this actual service.

Re: First Impressions with GPT-4V(ision)

#76
post #10

Sure, there are a few edge-case failures and mistakes here and there, but I can't help but be in awe . AWE. Let me state the obvious, in case anyone here isn't clear about the implications: If the rate of improvement of these AI models continues at the current pace, they will become a superior user interface to almost every thing you want to do on your mobile phone, your tablet, your desktop computer, your car, your…

Generally agree. Just to play devils advocate: If you want something done right, sometimes you have to do it yourself. Employees are sort of a universal UI. But you will always know more about what you want done than your agent, whether it’s human or computer. That’s even before considering the principal agent problem.

Just to play double-devils advocate:

If you want something done right, other times you will have to get someone else to do it. You know what you want, but you might not have the skills to do it. I can't represent myself well in court, do a good job of plumbing or cut my own hair, so I would ask for experts to do that for me.

Plus if someone is capable, it's often quicker to delegate than do, and if you are delegating to someone with more time to do the task they can often do a better job. Delegating unambiguously is a skill in itself, as instructing AIs will be.

Re: First Impressions with GPT-4V(ision)

#77
post #47

The discrepancy between the two answers regarding the set of coins is jarring. From the answer to the first question, one would assume that it can’t tell the currency. The answer to the second question shows that it actually can. The fact that LLMs don’t reflect a consistent inner model in that way, and hence the users’ inability to adequately reason about their AI interlocutor, is currently a severe usability issue.

>The fact that LLMs don’t reflect a consistent inner model in that way

You're probably not going to ask any human a question about an image and get every single detail you want every time. If you care about a detail, just ask about it. Doesn't really have anything to do with a consistent inner model.

Re: First Impressions with GPT-4V(ision)

#78

Earlier quoted context omitted.

There's enough vision-impaired people in the world to equal the population of Japan, Korea, and Vietnam combined. And beyond those people who would get obvious utility, this is essentially Google Lens on steroids—I simply can't figure how you could call this "niche". Maybe you won't use it multiple times per day, but plenty of people will. Hell, just now I was wondering why the leaves on one of my plants are starting…

Many of those people would be adequately served with a pair of glasses. This , wouldn't help them, even if they had both a device capable of using it and the means to pay for it. Even if it could help people, it's an open question if it would be safe, to, for example, use this to scan medication when it is only a probabilistic model that may hallucinate something that isn't actually there. What you're talking about i…

>What you're talking about is a speculative use of a service that might one day exist based on this technology.

It's not a speculative service that might one day happen.

Literally it's rolling out right now

https://www.bemyeyes.com/blog/announcing-be-my-ai

Re: First Impressions with GPT-4V(ision)

#79
post #29

Earlier quoted context omitted.

>> Who holds their phone up and takes a photo then wants to know it was a photo of? >> Great for the vision impaired. Yes, this is great for the estimated 285 million vision impaired people around the world[1]. [1] https://www.bemyeyes.com/about

Did you read my comment? I literally said that it’s for vision impaired. That’s great. …but it’s niche . I’m sitting on my couch right now and I can think of like 20 things I could chat to chatgpt about. I can see literally nothing in my visual range want to take a photo of and run image analysis over. It’s like Shazam. Yes, it’s useful, but, most of the time, I don’t need it. I would argue this is true for this, for…

You are thinking too small here. Think about a system that could provide you with sophisticated image analysis to the level of a human expert. Tumor identification, quality control, etc. I don't think we are there quite yet, but this is clear and fast progress on that field.

Re: First Impressions with GPT-4V(ision)

#80

I’m impressed, technically, but this seems niche. Who holds their phone up and takes a photo then wants to know it was a photo of? That’s weird. If you don’t know what it is, wtf did you take photo? The obvious use here is natural language improvement / photo editing for photos, but this is just a stepping stone to that, and bluntly, as it stands… the examples really don’t shine… Great for the vision impaired. …not s…

I'm bookmarking this and filing it alongside the infamous hacker news dropbox comment: https://news.ycombinator.com/item?id=9224

After all, an agent that is able to see, listen and communicate with a user with a superintelligent grasp of the world's information. Niche at best!
Post reply on HN