Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use in Gemini 3.5 Flash

blog.google

Post reply on HN
Print view View original

81–90 of 175 posts

Re: Computer use in Gemini 3.5 Flash

#81

Post by arjunchint »

Pretty doubtful about computer use/screenshotting based approaches.

With Retriever AI, we construct custom accessibility trees to represent web pages and just switched over to using DeepSeek v4 Flash and its nearing 100x cost decrease.

We also had great success just reverse engineering the underlying APIs of websites and then writing code to hit them. This approach of using screenshots to take actions on a webpage to trigger the underlying network calls the website is making seems too naive.

Re: Computer use in Gemini 3.5 Flash

#82

Post by airstrike »

Rebelgecko wrote: post #55
gdudeman wrote: post #26

Earlier quoted context omitted.

Computer use is a great idea. It gets the job done when nothing else will. If you're a person trying to get their job done at a big company, but half your job is in 1-2 proprietary tools or is stuck behind an API you can't program against, computer use can allow you, a non-techie, to do your job more efficiently. I think it's an awesome way to circumvent gate keepers and the IT department to let people accomplish the…

I think there's a sweet spot- a lot of the time you're probably better off with "reverse engineer this web page and build me an API or personalized chrome extension to meet my needs". I have an agent doing price checks for me for an item on a certain website. Instead of blasting through a zillion tokens processing the DOM over and over, it loaded the page once and figured out how to download a json with the price.

Does it have to view the page now repeatedly to download the JSON?

Re: Computer use in Gemini 3.5 Flash

#83

Post by airstrike »

Wowfunhappy wrote: post #76
orbital-decay wrote: post #38

Earlier quoted context omitted.

UI QA only works well if your model plausibly matches the average user behavior and/or real-world edge cases. These models are far from that, and they are much less random than you'd like them to be for fuzzing (mode collapse).

It doesn't need to be that kind of QA. Even just a basic "I want the AI to build the beginnings of a GUI app for me" will work much better if the AI can see the output of its work and iterate on it. Similar if you want the AI to fix a GUI bug—much better if you can show it the the bug and tell it how to test to see when it's gone.

the LLM does not require computer use to see the GUI and, again, that's a pretty niche use and not what Computer Use is being marketed for

Re: Computer use in Gemini 3.5 Flash

#84

Post by namuol »

sheept wrote: post #7
mlmonkey wrote: post #5

It's funny how in their own graph, https://storage.googleapis.com/gweb-uniblog-publish-prod/ima... Gemini 3.5 Flash is beat hands down by both Opus 4.8 and GPT 5.5, and yet the graph is drawn as if Gemini wins ... :-D

It highlights the Gemini models blue since that's what the article is about. The bar heights seem consistent with the values.

They should be sorting the models by performance on the horizontal axis.

Re: Computer use in Gemini 3.5 Flash

#85

Post by pimeys »

reacharavindh wrote: post #48
gdudeman wrote: post #26

Earlier quoted context omitted.

Computer use is a great idea. It gets the job done when nothing else will. If you're a person trying to get their job done at a big company, but half your job is in 1-2 proprietary tools or is stuck behind an API you can't program against, computer use can allow you, a non-techie, to do your job more efficiently. I think it's an awesome way to circumvent gate keepers and the IT department to let people accomplish the…

How are folks using “computer use” to click things on intranet portals that are behind an SSO? Even this OP example shows visitors a url and enter this search term… that is port of useless. How can I automate things behind an SSO wall? Even if it means I manually authorize it once and watch it do things on its own..

Take manual control once, save the login info to a password manager, teach the model to login with it.

Re: Computer use in Gemini 3.5 Flash

#86

Post by infecto »

arjunchint wrote: post #81

Pretty doubtful about computer use/screenshotting based approaches. With Retriever AI, we construct custom accessibility trees to represent web pages and just switched over to using DeepSeek v4 Flash and its nearing 100x cost decrease. We also had great success just reverse engineering the underlying APIs of websites and then writing code to hit them. This approach of using screenshots to take actions on a webpage to…

Reverse engineering APIs is just a recipe to get blocked sooner. Good luck!

Re: Computer use in Gemini 3.5 Flash

#87

Post by suuuuuuuu »

smallstepforman wrote: post #60

Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”. Stunned to…

I envy you that it admitted that rather than simply making up data and lying about it.

Re: Computer use in Gemini 3.5 Flash

#88

Post by YuechenLi »

So... has Google provided a Codex/Claude Code equivalent to Gemini yet? I would like to use Gemini for coding tasks, but that's kind of difficult to do as I don't even know how to get Gemini to even "clone this repo and read the code in it for static analysis", much less open PRs in repos.

ChatGPT/Codex can do it, Claude can do it, why can't Gemini?

And no, I don't mean going through Antigravity, and personally I'm wary about LLMs having unsupervised access on my computer without explicit policy, so I really think Google is putting the cart before the horse here.

Re: Computer use in Gemini 3.5 Flash

#89

Post by IncreasePosts »

fridder wrote: post #24

I wonder if it will be better at building TUI's. It has been absolutely abysmal at interacting with them and building them

That's my experience too. I've had increased luck encouraging the LLM to structure the code in "functional core, imperative shell" style, and telling it stupid things like "make sure you can test the code you're writing".

Re: Computer use in Gemini 3.5 Flash

#90

Post by jatora »

YuechenLi wrote: post #88

So... has Google provided a Codex/Claude Code equivalent to Gemini yet? I would like to use Gemini for coding tasks, but that's kind of difficult to do as I don't even know how to get Gemini to even "clone this repo and read the code in it for static analysis", much less open PRs in repos. ChatGPT/Codex can do it, Claude can do it, why can't Gemini? And no, I don't mean going through Antigravity, and personally I'm w…

no. gemini's instruction following is currently abysmal. Gemini CLI could be a great scaffold for all we know, but we cant know because the models it uses are so horribly bad at being driven in that way.

no clue why google has dropped the ball this hard on IF.

Post reply on HN