Google claims SoTA but it appears that, according to Apple, they may already be out of date: https://arxiv.org/abs/2404.05719
ScreenAI: A visual LLM for UI and visually-situated language understanding
11–20 of 44 posts
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#12is this similar to apple’s Realm?
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#13Google claims SoTA but it appears that, according to Apple, they may already be out of date: https://arxiv.org/abs/2404.05719
The core aspects of this research, datasets, and use cases discussed here have been in progress for quite a long time at Google (it's been WIP for many many years). The same can probably be said of Apple's paper though!
Congrats to all the folks involved :)
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#14How does this compare to the new GPT-4-turbo vision or Claude 3 Opus vision? Also, is this open source or can we access it with Vertex AI?
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#15I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.
huh exactly what I thought when I saw this
Isn't creating dataset for this the most easiest? we have source text of html and how they are rendered with all the intermediate info about tags, css layout etc available from most modern browsers.
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#16I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.
"Remove the ads from this page"
I can't wait for AI to become the ultimate ad-removal tool.
There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform).
There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users will accept a hamstrung interface that forces a subpar experience.
We basically just need to make sure Google, Microsoft/OpenAI, or some other industry giant doesn't win or that we don't wind up living under a cabal of just a few players.
I'm already hopefully imagining AI agents working for us to not just remove advertising noise, but to actively route around all of the times and places we're taken advantage of. That would be an excellent future.
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#17I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.
"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…
“It looks like this entire article is an advertorial piece for a book. Would you still like to read it?”
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#18I was looking for something similar recently and had found CogAgent[0] that looks quite interesting, has anyone tried anything similar? 0. https://github.com/THUDM/CogVLM?tab=readme-ov-file#gui-agent...
I haven't read through it yet, but there's FerretUI from Apple (mobile-specific, but I think a lot of learnings are generic) https://arxiv.org/abs/2404.05719
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#19At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding.
Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#20How does this compare to the new GPT-4-turbo vision or Claude 3 Opus vision? Also, is this open source or can we access it with Vertex AI?
We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress).
Unfortunately we can't compare it to ScreenAI directly since as far as I can tell it is not generally available. However ScreenAI does not appear to use a separate segmentation step, which we needed to implement in order to get good results.