ScreenAI: A visual LLM for UI and visually-situated language understanding • Hacker News Board

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#41

Post by rcthompson » Wed, Apr 10, 2024, 4:56 PM UTC

_boffin_ wrote: post #26

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

> Makes you wonder if the goal of their captcha system was ever really to stop people from botting. I don't think that was the main goal, but rather for them to get a massive labeling dataset for training their models on the cheap.

In turns out it's fine to have the snake eat its own tail if your real goal is just to keep the snake fed.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#42

Post by sebastiennight » Wed, Apr 10, 2024, 5:29 PM UTC

echelon wrote: post #16

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…

From time to time I see this comment pop up.

I think either I'm crossing paths with you a lot, and am always stricken by so much trusting enthusiasm... - or there're many people who have the same dream you're mentioning, in which case, one of you should build this product :-)

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#43

Post by abrichr » Thu, Apr 11, 2024, 3:20 PM UTC

williamdelo32 wrote: post #39

abrichr wrote: post #19

At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610

Interesting. Is this relying on SAM to segment text to send to GPT? How does it perform compared to GPT-V?

SAM is used to segment GUI elements, which are then sent to GPT4-V to be described. We then prompt GPT-4 with the user's actions and the descriptions of the GUI elements to generate new actions to accomplish different behaviors.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#44

Post by abrichr » Thu, Apr 11, 2024, 6:54 PM UTC

abrichr wrote: post #20

ilaksh wrote: post #14

How does this compare to the new GPT-4-turbo vision or Claude 3 Opus vision? Also, is this open source or can we access it with Vertex AI?

We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress). Unfortunately we can't compare it to ScreenAI directly since as far as I can tell it is not generally available. However ScreenAI does not appear to use a separate segmentation step, which we nee…

Update: it looks like Anthropic now accepts Canadian credit cards.

The results are not as good as GPT4-V or Gemini. I've posted the output for each of `gpt-4-vision-preview`, `gpt-4-turbo-2024-04-09`, `gemini-1.5-pro-latest`, and `claude-3-opus-20240229`. Claude is the only one who makes mistakes, at least on that test.