Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer Use is 45x more expensive than structured APIs

reflex.dev

Post reply on HN
Print view View original

51–60 of 295 posts

Re: Computer Use is 45x more expensive than structured APIs

#51

Post by gbriel »

merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

This is a good solution, instead of everyone blowing tokens on repeating the same computer use task, come up with a way to share the workflows. I think you'd need to make sure there aren't workflows shared that extract user information (passwords).

Re: Computer Use is 45x more expensive than structured APIs

#52

Post by kaashif »

donaldjbiden wrote: post #49
jackphilson wrote: post #45

Earlier quoted context omitted.

because the social media sites that do will outcompete once people get personal AI coaches that tell them to use technology that is better for them.

How is an AI posting on your social media better for you?

It's not, but token peddlers will say it is. It's good to interact with everything through buying tokens.

Re: Computer Use is 45x more expensive than structured APIs

#53

Post by awongh »

aurareturn wrote: post #8

In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.

At the beginning of the internet we were promised the free flow of digital information between computers, peer-to-peer. What we got was silos of content each fighting each other to make sure that the silos stay intact with DRM.

I could imagine an AI future where agentic shopping companies who promise me the best deal are pitted against Walmart and Amazon, trying to algorithmically squeeze me for $2 more- just two bots playing a cat and mouse game to save me a few bucks.

For some reason a lot of tech ends up in these antagonistic monopolies- Apple wants to sell privacy aware devices as a product feature, Google wants give you mail and maps, but sell your data. Despite any appearances neither give a shit about you, even if you benefit from the dynamic.

Re: Computer Use is 45x more expensive than structured APIs

#54

Post by FirestarAlpha »

aurareturn wrote: post #8

In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.

That’s actually what the Reflex plugin behind the APIs in the benchmark does. It creates APIs from your app’s event handlers, thereby providing a stateful way for agents to navigate apps.

It’s why we did this benchmark :) - reflex team member

Re: Computer Use is 45x more expensive than structured APIs

#55

Post by merlindru »

gbriel wrote: post #51
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

This is a good solution, instead of everyone blowing tokens on repeating the same computer use task, come up with a way to share the workflows. I think you'd need to make sure there aren't workflows shared that extract user information (passwords).

this is protected against at the OS level, provided the applications declare the input correctly as a SecureTextField.

i so far haven't found any application that doesn't.

all you're able to get out, as far as i can tell, is the length of the entered password.

Re: Computer Use is 45x more expensive than structured APIs

#57

Post by moralestapia »

palashawas wrote: post #24
moralestapia wrote: post #11

This is obvious. The problem is that not everything has an API, while everything has a human-oriented UI.

Right - we did this benchmark because we launched a plugin that makes APIs programmatically from an app's human-oriented UI (from the event handlers, to be specific). So any app that has a human-oriented UI now has an API. The benchmark is a more generally interesting part of the launch materials, so I figured it had its own separate home here.

That is actually great, I'll definitely check it out. Thanks!

Re: Computer Use is 45x more expensive than structured APIs

#58

Post by teej »

merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

You should call it Braille

Re: Computer Use is 45x more expensive than structured APIs

#59

Post by merlindru »

teej wrote: post #58
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

You should call it Braille

shit, why didn't i think of that

i tend to think of invoke as "an API over macOS apps" tho...

doesn't `invoke finder shareAndCopyLink` read very nicely? :P

Re: Computer Use is 45x more expensive than structured APIs

#60

Post by peyton »

overgard wrote: post #37

I've been thinking of things I'd want an agent for recently. The problem is, everything I think of is something that requires using quite a few different websites, saving a lot of data securely, and working with a lot of sensitive accounts (my email, etc.) The problem is, all the tasks are essentially: a) things agents probably just can't do, and b) things that absolutely cannot afford to be hallucinated or otherwise…

It’s great at

1. things you wouldn’t otherwise bother doing

2. things where it otherwise would get stuck iterating on hacky workarounds doomed to fail

“Reverse engineer this app/site so we can do $common_task in one click”, “by the way, I’m logged in to $developer_portal, so try @Browser Use if you’re stuck”, etc.

I just had Codex pull user flows out of a site I’m working on and organize them on a single page. It found 116. I went in and annotated where I wanted changes, and now it’s crunching away fixing them all. Then it’ll give me an updated contact sheet and I can do a second pass.

I’d never do this sort of quality pass manually and instead would’ve just fixed issues as they came up, but this just runs in the background and requires 15 minutes of my time for a lot of polish.

Post reply on HN