doesn't seem like it makes sense to train AI around human user interfaces which aren't really efficient. It is like building a mechanical horse.
If we could build mechanical horses they wiuld be absolutely amazing!
Gemini 2.5 Computer Use model
61–70 of 339 posts
Re: Gemini 2.5 Computer Use model
#62[dead]
Re: Gemini 2.5 Computer Use model
#63Interesting, seems to use 'pure' vision and x/y coords for clicking stuff. Most other browser automation with LLMs I've seen uses the dom/accessibility tree which absolutely churns through context, but is much more 'accurate' at clicking stuff because it can use the exact text/elements in a selector.
Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.
Re: Gemini 2.5 Computer Use model
#64[dead]
Re: Gemini 2.5 Computer Use model
#65I assume its tool calling and structured output are way better, but this model isn't in Studio unless its being silently subbed in.
Just tried it in an existing coding agent and it rejected the requests because computer tools weren't defined.
We can definitely make the docs more clear here but the model requires using the computer_use tool. If you have custom tools, you'll need to exclude predefined tools if they clash with our action space.
See this section: https://googledevai.devsite.corp.google.com/gemini-api/docs/...
And the repo has a sample setup for using the default computer use tool: https://github.com/google/computer-use-preview
Re: Gemini 2.5 Computer Use model
#66Hey - I'm on the team that launched this. Please let me know if you have any questions!
How are you going to deal with reCAPTCHA and ad impressions? Sounds like a conflict of interest.
Re: Gemini 2.5 Computer Use model
#67It successfully got through the captcha at https://www.google.com/recaptcha/api2/demo
Post edited: I was wrong about this. Gemini tried to solve the Google CAPTCHA but it was actually Browserbase that did the solve, notes here: https://simonwillison.net/2025/Oct/7/gemini-25-computer-use-...
Interesting that they're allowing Gemini to solve CAPTCHAs because OpenAI's agent detects and forces user-input for CAPTCHAs despite being fully able to solve them
Re: Gemini 2.5 Computer Use model
#68Computer Use models are going to ruin simple honeypot form fields meant to detect bots :(
I just tried to submit a contact form with it. It successfully solved the ReCaptcha but failed to fill in a required field and got stuck. We're safe.
Re: Gemini 2.5 Computer Use model
#69Earlier quoted context omitted.
What sorts of automations were you able to get working with the Chrome dev tools MCP?
Not OP, but in my experience, Jest and Playwright are so much faster that it's not worth doing much with the MCP. It's a neat toy, but it's just too slow for an LLM to try to control a browser using MCP calls.
Yeah I think it would be better to just have the model write out playwright scripts than the way it's doing it right now (or at least first navigate manually and then based on that, write a playwright typescript script for future tests).
Cuz right now it's way too slow... perform an action, then read the results, then wait for the next tool call, etc.
Re: Gemini 2.5 Computer Use model
#70This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance. Obviously much harder with UI vs agent events similar to the below. https://docs.claude.com/en/docs/claude-code/hooks https://google.github.io/adk-docs/callbacks/
Hi! I work in identity products at Browserbase. I’ve spent a fair amount of time lately thinking about how to layer RBAC across the web. Do you think callbacks are how this gets done?
Disclaimer: Im a cofounder, we focus critical spaces with AI. Also i was the feature request for claude code hooks.
But my bet - we will not deploy a single agent into any real environment without deterministic guarantees. Hooks are a means...
Browserbase with hooks would be really powerful, governance beyond RBAC (but of course enabling relevant guardrailing as well - "does agent have permission to access this sharepoint right now, within this context, to conduct action x?").
I would love to meet with you actually, my shop cares intimately about agent verification and governance. Soon to release the tool I originally designed for claude code hooks.