Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ScreenAI: A visual LLM for UI and visually-situated language understanding

research.google

Post reply on HN
Print view View original

21–30 of 44 posts

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#21

Post by Xenoamorphous »

echelon wrote: post #16
S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…

Everything behind a paywall then?

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#22

Post by notahacker »

spxneo wrote: post #8
S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

that would be the 4d chess move: imagine when you get a captcha with "click 1 thing from things that do not fly" but you actually helping select drone targets somewhere in middle east

this is what happens at the intersection of unlimited VC money for "AI" and wannabe entrepreneurs that read Ender's Game and thought "business opportunity here"

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#23

Post by spxneo »

abrichr wrote: post #19

At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610

MIT license too (getting rare)! mad respects, thank you.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#24

Post by spxneo »

abrichr wrote: post #20
ilaksh wrote: post #14

How does this compare to the new GPT-4-turbo vision or Claude 3 Opus vision? Also, is this open source or can we access it with Vertex AI?

We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress). Unfortunately we can't compare it to ScreenAI directly since as far as I can tell it is not generally available. However ScreenAI does not appear to use a separate segmentation step, which we nee…

can you elaborate on "extremely well"? where is it currently falling short?

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#25

Post by EZ-Cheeze »

Imagine a top-level screen filter that processes all you see in ways which you define. For example, "hide all faces" can help you spot new details in a movie since your eyes won't be automatically attracted to faces. Or "hide all proper names" can make internet browsing more interesting and mysterious

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#26

Post by _boffin_ »

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

> Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

I don't think that was the main goal, but rather for them to get a massive labeling dataset for training their models on the cheap.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#27

Post by taneq »

notduncansmith wrote: post #17
echelon wrote: post #16

Earlier quoted context omitted.

"Remove the ads from this page" I can't wait for AI to become the ultimate ad-removal tool. There might be an arms race, but the anti-ad side will win as long as there isn't a unilateral winner (strongest models, biggest platform). There will be enough of a shake up to the current regime -- search, browsers, etc. -- that there is opportunity for new players to attack multiple fronts. Given choice, I don't think users…

“It looks like this entire article is an advertorial piece for a book. Would you still like to read it?”

And most people would say it’s broken and only returns false positives. Well, half right, but the positives aren’t false…

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#28

Post by tjtang2019 »

I think the interpretability and privacy protection of the model are also issues that need attention, especially in scenarios involving user personal data and privacy. Therefore, although this research result has potential, further research and exploration are still needed in practical applications.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#29

Post by abrichr »

spxneo wrote: post #23
abrichr wrote: post #19

At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610

MIT license too (getting rare)! mad respects, thank you.

Thank you!

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#30

Post by abrichr »

spxneo wrote: post #24
abrichr wrote: post #20

Earlier quoted context omitted.

We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress). Unfortunately we can't compare it to ScreenAI directly since as far as I can tell it is not generally available. However ScreenAI does not appear to use a separate segmentation step, which we nee…

can you elaborate on "extremely well"? where is it currently falling short?

You can see how it performs in describing GUI elements in the linked PR if you scroll down -- here's a direct link:

https://private-user-images.githubusercontent.com/774615/320...

Regarding failure modes, we have yet to do extensive testing, but I've seen it confuse the divide and subtract buttons on the calculator before only once.

Post reply on HN