Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer Use is 45x more expensive than structured APIs

reflex.dev

Post reply on HN
Print view View original

101–110 of 295 posts

Re: Computer Use is 45x more expensive than structured APIs

#101

Post by etothet »

Vision has a long way to go. I remember trying an early version of AWS's Nova Act and laughed at how slow it was. And a few months later it hadn't really seemed to improve that much.

Recently, I asked Claude to log into my local grocery store chain's website and add all of the items from my shopping list to a cart. It was hilariously slow, but it did get the job done.

Unless I missed it, the article doesn't explictly mention speed in the copy, but the results do show a 17 minute (!!!) total time for the vision agent vs. 0.5s - 2.8s for the API approach.

A big part of the challenge with vision is that to manipulate the DOM, you first have to be sure the entire (current) DOM is loaded. In my experience this ends up in adding a lot of artificial waits for certain elements to exist on the page.

Re: Computer Use is 45x more expensive than structured APIs

#102

Post by tomjakubowski »

ctoth wrote: post #72
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

If agents is what it finally takes to get good a11y I'll take it. I'll bitch about it, but I'll take it.

Playwright, the end-to-end testing framework for the web, provides a strong incentive to give sites good a11y: Playwright tests are an absolute delight to read, write and maintain on properly accessible sites, when using the accessibility locators. Somewhat less so when using a soup of CSS selector and getByText()-style locators.

One thing I am curious about is a hybrid approach where LLMs work in conjunction with vision models (and probes which can query/manipulate the DOM) to generate Playwright code which wraps browser access to the site in a local, programmable API. Then you'd have agents use that API to access the site rather than going through the vision agents for everything.

Re: Computer Use is 45x more expensive than structured APIs

#103

Post by boringg »

rahulyc wrote: post #98

All the websites currently blocking Claude Code or other AI agents are fighting a losing battle. Computer-use is in the early stages, and the thing preventing mass-adoption seems to be the number of tokens it takes. Agents can fumble around trying 10 CLI commands that don't work before finding the right one and we barely notice. But other visual agents (browser use / computer use etc) end up eventually fumbling on to…

Tokens cheaper? I don't think that seems to be the case ... VC funded tokens were there to build user base and token price will go up as they eventually switch from growth to profitability.

Re: Computer Use is 45x more expensive than structured APIs

#104

Post by bheadmaster »

boringg wrote: post #103
rahulyc wrote: post #98

All the websites currently blocking Claude Code or other AI agents are fighting a losing battle. Computer-use is in the early stages, and the thing preventing mass-adoption seems to be the number of tokens it takes. Agents can fumble around trying 10 CLI commands that don't work before finding the right one and we barely notice. But other visual agents (browser use / computer use etc) end up eventually fumbling on to…

Tokens cheaper? I don't think that seems to be the case ... VC funded tokens were there to build user base and token price will go up as they eventually switch from growth to profitability.

It will take a few years until scheduled data center construction finishes, and together with software optimizations that may come up in the meantime, it may cause a significant decrease in token price.

Re: Computer Use is 45x more expensive than structured APIs

#105

Post by johnsmith1840 »

rahulyc wrote: post #98

All the websites currently blocking Claude Code or other AI agents are fighting a losing battle. Computer-use is in the early stages, and the thing preventing mass-adoption seems to be the number of tokens it takes. Agents can fumble around trying 10 CLI commands that don't work before finding the right one and we barely notice. But other visual agents (browser use / computer use etc) end up eventually fumbling on to…

And the lethal trifecta but I suppose that's all agents as of now anyhow. Every AI provider has major warnings about letting AI have access to PII in the browser.

Re: Computer Use is 45x more expensive than structured APIs

#106

Post by Aurornis »

boringg wrote: post #103
rahulyc wrote: post #98

All the websites currently blocking Claude Code or other AI agents are fighting a losing battle. Computer-use is in the early stages, and the thing preventing mass-adoption seems to be the number of tokens it takes. Agents can fumble around trying 10 CLI commands that don't work before finding the right one and we barely notice. But other visual agents (browser use / computer use etc) end up eventually fumbling on to…

Tokens cheaper? I don't think that seems to be the case ... VC funded tokens were there to build user base and token price will go up as they eventually switch from growth to profitability.

I wish I could place a lot of money on the opposite side of this bet.

I don't think many realize how could the cheap, alternative models are becoming. I prefer SOTA models for key work, but I can also spend 10X as many tokens on an open model hosted by a non-VC subsidized provider (who is selling at a profit) for tasks that can tolerate slightly less quality.

The situation is only getting better as models improve and data centers get built out.

Re: Computer Use is 45x more expensive than structured APIs

#107

Post by angry_octet »

Worf wrote: post #90

Is it possible to ask the vision agent to "map" the UI and expose it to another agent as a set of interfaces that resemble an API better? From what I understand the vision agent now should both know that "next page" shows more results and that they need to get more results in the first place. If one agent just explores the UI, maybe in a test environment, and outputs a somewhat-structured description of the various U…

I think you're right, you can get agents to do what we do -- learn how a website works. Then expose that model as a simple API. There will still be some vision tasks for navigation but they will be just vision tasks, no thinking required.

Re: Computer Use is 45x more expensive than structured APIs

#108

Post by lopis »

merlindru wrote: post #75
ctoth wrote: post #72

Earlier quoted context omitted.

If agents is what it finally takes to get good a11y I'll take it. I'll bitch about it, but I'll take it.

i think this goes both ways too :) agents have been a boon for everyone with disabilities, carpal tunnel, RSI, ADHD, anything and now the fact that interfaces need to be accessible to agents, not just humans, ironically increases it for humans in return

And lets not forget that not all disabilities are chronic. Many disabilities are situational or temporary. AI is a great assist for a hangover day for example...

Re: Computer Use is 45x more expensive than structured APIs

#109

Post by JambalayaJimbo »

aurareturn wrote: post #77
planb wrote: post #23

Earlier quoted context omitted.

This will not happen. None of the existing apps people use daily on their phones have any incentive to support this. Social media wants the people to doomscroll, shopping apps and booking sites want to use their own dark patterns to make people believe they get a special discount if they buy _now_ and everything else just wants users to see the ads. Why on earth would they offer convenient hooks for AI chatbots?

Why on earth would they offer convenient hooks for AI chatbots? Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more. Watch how fast Meta adds this if a new hot shot social media app succeeds by designing for AI agents controlled by users.

>Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more.

This is the exact opposite of what will happen (and in fact what has happened). Reddit is suing Perplexity right now for scraping.

Meta will not serve content to some other app for free - for what benefit? They will not see advertising data.

Re: Computer Use is 45x more expensive than structured APIs

#110

Post by boringg »

Aurornis wrote: post #106
boringg wrote: post #103

Earlier quoted context omitted.

Tokens cheaper? I don't think that seems to be the case ... VC funded tokens were there to build user base and token price will go up as they eventually switch from growth to profitability.

I wish I could place a lot of money on the opposite side of this bet. I don't think many realize how could the cheap, alternative models are becoming. I prefer SOTA models for key work, but I can also spend 10X as many tokens on an open model hosted by a non-VC subsidized provider (who is selling at a profit) for tasks that can tolerate slightly less quality. The situation is only getting better as models improve and…

Fair - there are bets both ways though I wouldn't consider it to be a certainty. That revenue drive on this AI build out is going to be real and multifold.
Post reply on HN