Computer Use is 45x more expensive than structured APIs
1–10 of 295 posts
Re: Computer Use is 45x more expensive than structured APIs
#2[deleted]
Re: Computer Use is 45x more expensive than structured APIs
#3The interface designed for humans is poor for AI needs? And the interface designed for programmatic use is easier for the AI to use? In other news, the sky is blue and water is wet.
Re: Computer Use is 45x more expensive than structured APIs
#4I'm pretty unsurprised that the vision agent did worse. I'd be interested in a comparison between the different tools that now exist to let LLMs drive browsers (e.g. vercel's agent-browser, the relatively new dev-browser[1], etc.)
There are usecases where the vision agent is the more obvious, or only choice though, e.g. prorprietary/locked-down desktop apps that lack an automation layer.
Re: Computer Use is 45x more expensive than structured APIs
#5I think of computer use as like last mile delivery. APIs and bash and such are the efficient logistics networks. Both have different benefits. Obviously, use the efficient methods when you can.
Re: Computer Use is 45x more expensive than structured APIs
#6The interface designed for humans is poor for AI needs? And the interface designed for programmatic use is easier for the AI to use? In other news, the sky is blue and water is wet.
Yep, everyone knows computer use is more expensive. This is about quantifying the gap
Re: Computer Use is 45x more expensive than structured APIs
#7> This is not a model problem. The vision agent was reasoning about a rendered page and had no signal that the page wasn't showing everything.
> To make the comparison apples-to-apples, we rewrote the vision prompt as an explicit UI walkthrough, naming the sidebar items, tabs, and form fields the agent should interact with at each step. Fourteen numbered instructions covering the navigation the agent had failed to figure out on its own.
This is a model problem, though. Because the model failed to understand it could scroll, you forced it to consume multiples of the tokens. Could you come up with an alternative here?
Do you know what the vision model was trained on? Because often people see “vision model” and think “human-level GUI navigator” when afaik the latter has yet to be built.
Re: Computer Use is 45x more expensive than structured APIs
#8In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly.
I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.
Re: Computer Use is 45x more expensive than structured APIs
#9In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.
Openai should not design a phone... They should try making money first
Re: Computer Use is 45x more expensive than structured APIs
#10I'm pretty unsurprised that the vision agent did worse. I'd be interested in a comparison between the different tools that now exist to let LLMs drive browsers (e.g. vercel's agent-browser, the relatively new dev-browser[1], etc.) There are usecases where the vision agent is the more obvious, or only choice though, e.g. prorprietary/locked-down desktop apps that lack an automation layer. 1. https://github.com/SawyerH…
Interesting! I'll play around with agent-browser and update this article if anything comes up