I can't wait for the new wave of terrible UIs specifically designed to fool AI agents into clicking the "send me all your money" button. (For bonus points, do this while making the UI seem perfectly reasonable to humans.)
ScreenAI: A visual LLM for UI and visually-situated language understanding
31–40 of 44 posts
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#32pardon my ignorance, but can I run this model locally?
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#33Imagine a new crop of QA automation tooling that's going to leverage these capabilities!
• Semantic change comparison between screenshots. Visual regression testing, where you prompt the model to ignore certain things instead of masking and where it labels the changes with a message, like "chart color changed" or "text shifted down by 3 pixels".
• Using plain English as test scenarios, instead of brittle WebDriver-like APIs.
• Autonomous agent fuzzing the application by free roaming the UI.
• RAG the design artifacts from Jira and Google Docs for more targeted feature exploration and test scenario generation.
• Automatic bug reports as the output of the above. Or even send a draft PR to fix an issue, while at it!
The more I think about use cases, the more it sounds like full software development automation. Late in game, we won't probably need software as it exists today at all. This feels like reading Accelerando again, but this time it's happening for real and to you.
P.S.: Didn't expect to see a cafe from Cyprus - Akakiko Limassol - used as the demo, I'll remember to visit next time I'm in the area :P
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#34I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.
"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…
I'm now envisioning an ad framework, where instead of selling rectangles of content, advertisers bid to use an llm to rewrite the whole article with at least {{4}} mentions of {{how refreshing pepsi is}}.
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#35I can't wait for the new wave of terrible UIs specifically designed to fool AI agents into clicking the "send me all your money" button. (For bonus points, do this while making the UI seem perfectly reasonable to humans.)
that would take UI dark patterns to a new level
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#36I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.
"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…
Apple won't allow AI-powered extensions. They currently don't even allow blacklist-powered extensions like UBlock.
And that's the "privacy" focussed company, shoving ads down your throat.
The competitor is Google.
And 90% of users spend 90% of their time in walled gardens like Instagram or TikTok anyway. They see built-in ads.
Do I need to say more?
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#37pardon my ignorance, but can I run this model locally?
I wondered that too, it's Google so it's going to be Tensorflow or accessible through some workbook.. and I've lost interest.
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#38As we were discussing recently how blind people use the computer, navigate the web, and write programs in code editors - ScreenAI and other ways of giving LLMs a visual mode are promising, giving people the ability to understand and interact with visual interfaces using natural language.
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#39At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610
Interesting. Is this relying on SAM to segment text to send to GPT? How does it perform compared to GPT-V?
Re: ScreenAI: A visual LLM for UI and visually-situated language understanding
#40Is the model being released?