Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer Use is 45x more expensive than structured APIs

reflex.dev

Post reply on HN
Print view View original

91–100 of 295 posts

Re: Computer Use is 45x more expensive than structured APIs

#91

Post by lelanthran »

Havoc wrote: post #16

Isn't it possible to somehow wire this into the window manager? Wayland or whatever. Have it speak the native window lang rather than crunch the pixels? At least for the majority. I can see the appeal in pixel route given universality but wow that seems ugly on efficiency

> Isn't it possible to somehow wire this into the window manager? Wayland or whatever. Have it speak the native window lang rather than crunch the pixels? At least for the majority.

Not possible on wayland, maybe on X11 protocol?

Re: Computer Use is 45x more expensive than structured APIs

#92

Post by RobRivera »

UX feedback

Me: hmm, this title confuses and infuriates Rob.

[Clicks link]

Me: Sees same title, repeat feelings of confusion and infuration

[Scrolls article down on my smartphone]

Me: Sees jpg with the same title, repeat feelings of co fusion and infuriation.

[Closes tab]

[Continues living rest of my life]

I hope this feedback is well received and understood.

Re: Computer Use is 45x more expensive than structured APIs

#94

Post by arjunchint »

The hard part about the web is that API's aren't just available even if the website owner wants them exposed (big if).

I embedded a Google Calendar widget on my Book a demo page, I don't know the API and Google doesn't expose/maintain one either.

What we are doing at Retriever AI is to instead reverse engineer the website APIs on the fly and call them directly from within the webpage so that auth/session tokens propoagate for free: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

Re: Computer Use is 45x more expensive than structured APIs

#95

Post by wahnfrieden »

ai_fry_ur_brain wrote: post #86
wahnfrieden wrote: post #79

Earlier quoted context omitted.

It’s not a fad or without value.

Its very much valuable to lazy people who dont care about quality or doing hard things. I totally see the appeal for those people.

Sounds like you are more interested in performativity / aesthetics of production if you think writing software in a harder way is an indisputable virtue just because it requires more effort. On top of that you are an elitist about it

Agent use can be used to improve quality and maintainability

Re: Computer Use is 45x more expensive than structured APIs

#96

Post by hellojimbo »

merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

Isn't that basically what browser base does. I've found the hardest part of browser use to be stealth first then client change management then browser comprehension (which gets better with every new model).

Re: Computer Use is 45x more expensive than structured APIs

#97

Post by ssl-3 »

aurareturn wrote: post #8

In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.

We'll just close the loop with a systemd MCP, set the shell to /usr/bin/codex, and find some other way to pay the bills.

Perfect.

Re: Computer Use is 45x more expensive than structured APIs

#98

Post by rahulyc »

All the websites currently blocking Claude Code or other AI agents are fighting a losing battle. Computer-use is in the early stages, and the thing preventing mass-adoption seems to be the number of tokens it takes. Agents can fumble around trying 10 CLI commands that don't work before finding the right one and we barely notice. But other visual agents (browser use / computer use etc) end up eventually fumbling on to the right thing, but we don't have the patience to wait 20 mins. to click a button. As tokens get cheaper + faster, we probably get the models that can use a UI interface just as natively as a CLI.

Re: Computer Use is 45x more expensive than structured APIs

#99

Post by johnsmith1840 »

Text based web browsing? Would love the comparison there. Tons of systems have a dom translation layer. I'm building around this with the concept of turn a webpage into text for an agent to use directly. I actually had to move away from haiku not because of accuracy problems but because it operated the browser too fast for a human to follow what it was doing. The real loss here are bespoke webapps like a figma or google docs which are near impossible to see what they are doing via the dom.

To me the browser is a translation layer. Working on the browser directly while hard enables big advantages on compatibility. The only thing I miss as of now which is on the todo is ocr of the images in the browser into text out. But an api would need to do that anyways to work.

The main loss in my view of pure API based is, where do you get the data? We won't replicate human work without seeing that done. Humans work in the UI that's it. Computer use to me is the promise of being able to replicate end to end actions a human does. API can do that in theory but the data to do that is also near impossible to collect properly.

Re: Computer Use is 45x more expensive than structured APIs

#100

Post by merlindru »

hellojimbo wrote: post #96
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

Isn't that basically what browser base does. I've found the hardest part of browser use to be stealth first then client change management then browser comprehension (which gets better with every new model).

i'm not too familiar with browserbase, but invoke works with any macOS app (or at least the accessible ones), i think browserbase is only for browser usage.

in the context of this blog post, the conclusion looks similar though!

"use the whole web like it's an API"

works much better than

"figure out similar or identical tasks from a clean slate every single time you do them"

Post reply on HN