Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Computer Use model

blog.google

Post reply on HN
Print view View original

111–120 of 339 posts

Re: Gemini 2.5 Computer Use model

#111

Post by omneity »

atonse wrote: post #69
odie5533 wrote: post #31

Earlier quoted context omitted.

Not OP, but in my experience, Jest and Playwright are so much faster that it's not worth doing much with the MCP. It's a neat toy, but it's just too slow for an LLM to try to control a browser using MCP calls.

Yeah I think it would be better to just have the model write out playwright scripts than the way it's doing it right now (or at least first navigate manually and then based on that, write a playwright typescript script for future tests). Cuz right now it's way too slow... perform an action, then read the results, then wait for the next tool call, etc.

This is basically our approach with Herd[0]. We operate agents that develop, test and heal trails[1, 2], which are packaged browser automations that do not require browser use LLMs to run and therefore are much cheaper and reliable. Trail automations are then abstracted as a REST API and MCP[3] which can be used either as simple functions called from your code, or by your own agent, or any combination of such.

You can build your own trails, publish them on our registry, compose them ... You can also run them in a distributed fashion over several Herd clients where we take care of the signaling and communication but you simply call functions. The CLI and npm & python packages [4, 5] might be interesting as well.

Note: The automation stack is entirely home-grown to enable distributed orchestration, and doesn't rely on puppeteer nor playwright but the browser automation API[6] is relatively similar to ease adoption. We also don't use the Chrome Devtools Protocol and therefore have a different tradeoff footprint.

0: https://herd.garden

1: https://herd.garden/trails

2: https://herd.garden/docs/trails-automations

3: https://herd.garden/docs/reference-mcp-server

4: https://www.npmjs.com/package/@monitoro/herd

5: https://pypi.org/project/monitoro-herd/

6: https://herd.garden/docs/reference-page

Re: Gemini 2.5 Computer Use model

#112

Post by enjoylife »

> It is not yet optimized for desktop OS-level control

Alas, AGI is not yet here. But I feel like if this OS-level of control was good enough, and the cost of the LLM in the loop wasn't bad, maybe that would be enough to kick start something akin to AGI.

Re: Gemini 2.5 Computer Use model

#114

Post by alganet »

enjoylife wrote: post #112

> It is not yet optimized for desktop OS-level control Alas, AGI is not yet here. But I feel like if this OS-level of control was good enough, and the cost of the LLM in the loop wasn't bad, maybe that would be enough to kick start something akin to AGI.

I am curious. Why do you think controlling an OS (and not just a browser) would be a move towards AGI?

Re: Gemini 2.5 Computer Use model

#115

Post by bonoboTP »

measurablefunc wrote: post #107
ionwake wrote: post #105

Earlier quoted context omitted.

Why are you talking about image processing ? The guy you’re talking to isn’t

What do you suppose "render" means?

The original comment I replied to said "You can navigate a website without visually decoding the image of a website." I replied that decoding is necessary to know where the elements will end up in a visual arrangement, because often that carries semantics. A label that is rendered next to another element can be crucial for understanding the functioning of the program. It's nontrivial just from the HTML or whatever tree structure where each element will appear in 2D after rendering.

Re: Gemini 2.5 Computer Use model

#116

Post by chrisfosterelli »

dekhn wrote: post #71

Many years ago I was sitting at a red light on a secondary road, where the primary cross road was idle. It seemed like you could solve this using a computer vision camera system that watched the primary road and when it was idle, would expedite the secondary road's green light. This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that…

Ironically now that computer vision is commonplace, the cameras you talk about have become increasingly popular over the years because the magnetic systems do not do a very good job of detecting cyclists and the cameras double as a congestion monitoring tool for city staff.

Re: Gemini 2.5 Computer Use model

#117

Post by bonoboTP »

measurablefunc wrote: post #110
bonoboTP wrote: post #103

Earlier quoted context omitted.

It would have to implicitly render the HTML+CSS to know which two elements visually end up next to each other, if the markup is spaghetti and badly done.

The linked post demonstrates arbitrary re-ordering of image patches. Spatial continuity is not relevant to neural networks.

That's ridiculous, sorry. If that were so, we wouldn't have positional encodings in vision transformers.

Re: Gemini 2.5 Computer Use model

#119

Post by alganet »

John7878781 wrote: post #56
TIPSIO wrote: post #29

Painfully slow

That doesn't matter so much when it can happen in the background.

It matters a lot for E2E testing. I would totally replace the ease of the AI solution for a faster, more complicated one if it starts impacting build times.

Few things are more frustrating for a team than maintaining a slow E2E browser test suite.

Re: Gemini 2.5 Computer Use model

#120

Post by measurablefunc »

bonoboTP wrote: post #117
measurablefunc wrote: post #110

Earlier quoted context omitted.

The linked post demonstrates arbitrary re-ordering of image patches. Spatial continuity is not relevant to neural networks.

That's ridiculous, sorry. If that were so, we wouldn't have positional encodings in vision transformers.

It's not ridiculous if you understand how neural networks actually work. Your perception of the numbers has nothing to do w/ the logic of the arithmetic in the network.
Post reply on HN