ScreenAI: A visual LLM for UI and visually-situated language understanding

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software.

Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#5

Post by warthog » Tue, Apr 09, 2024, 6:40 PM UTC

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

huh exactly what I thought when I saw this

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#6

Post by [deleted] » Tue, Apr 09, 2024, 6:54 PM UTC

namanyayg wrote: post #3

I was looking for something similar recently and had found CogAgent[0] that looks quite interesting, has anyone tried anything similar? 0. https://github.com/THUDM/CogVLM?tab=readme-ov-file#gui-agent...

[deleted]

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#7

Post by piecerough » Tue, Apr 09, 2024, 6:54 PM UTC

"We are also releasing three new datasets: Screen Annotation to evaluate the layout understanding capability of the model, as well as ScreenQA Short and Complex ScreenQA for a more comprehensive evaluation of its QA capability."

Looks useful to me for replicating some things. Good stuff!

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#8

Post by spxneo » Tue, Apr 09, 2024, 6:57 PM UTC

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

that would be the 4d chess move: imagine when you get a captcha with "click 1 thing from things that do not fly" but you actually helping select drone targets somewhere in middle east

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#9

Post by [deleted] » Tue, Apr 09, 2024, 6:58 PM UTC

piecerough wrote: post #7

"We are also releasing three new datasets: Screen Annotation to evaluate the layout understanding capability of the model, as well as ScreenQA Short and Complex ScreenQA for a more comprehensive evaluation of its QA capability." Looks useful to me for replicating some things. Good stuff!

[deleted]

Re: ScreenAI: A visual LLM for UI and visually-situated language understanding

#10

Post by htrp » Tue, Apr 09, 2024, 7:00 PM UTC

spxneo wrote: post #8

S0y wrote: post #4

I find it quite ironic that google are the biggest players in creating solutions that actively contribute in defeating their very own anti automation software. Makes you wonder if the goal of their captcha system was ever really to stop people from botting.

that would be the 4d chess move: imagine when you get a captcha with "click 1 thing from things that do not fly" but you actually helping select drone targets somewhere in middle east

https://xkcd.com/1897/