Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer Use is 45x more expensive than structured APIs

reflex.dev

Post reply on HN
Print view View original

221–230 of 295 posts

Re: Computer Use is 45x more expensive than structured APIs

#221

Post by [deleted] »

zmmmmm wrote: post #163
angry_octet wrote: post #111

Great guidance hidden in here for making it expensive for agents to navigate your website. Move elements on screen as the mouse moves, force natural mouse movement to make the UI work, change the button labels in the JS to be randomly named every visit, force scrolling to the bottom of the screen to check for hidden extra tasks... Hang on, that sounds like common corporate SaaS apps.

It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perceive…

[deleted]

Re: Computer Use is 45x more expensive than structured APIs

#222

Post by willwade »

merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

Interesting! I started something - nowhere near as complete as that and quite different but again using accessibility UI elements. The BIG problem I've found is SOOOO much stuff does really poorly having these elements exposed. Here was my approach https://github.com/willwade/app-automate?tab=readme-ov-file#... - What I do here is build UI templates - either using UIAccess OR using a one pass using a vision model.

Now the argument against this on [reddit](https://www.reddit.com/r/openclaw/comments/1s1dzxq/comment/o...)

"my experience is the opposite actually. UIA looks uniform on paper but WPF, WinForms, and Win32 all expose different control patterns and you end up writing per-toolkit handlers anyway. Qt only exposes anything if QAccessible was compiled in and the accessibility plugin is loaded at runtime, which on shipped binaries is basically never. Electron is just as opaque on Windows as on macOS because it's the same chromium underneath drawing into a canvas. the real split isn't OS vs OS, it's native toolkit vs everything else."

Re: Computer Use is 45x more expensive than structured APIs

#223

Post by willwade »

btown wrote: post #143
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

If you're on macOS and interested in this space, I highly recommend you open up the system-provided Accessibility Inspector.app and play around with apps and browsers. See how the green cells might guide an LLM to only need to read/OCR specific parts of a screen, how much text is already natively available to the accessibility engine, and how this could lead to really effective hybrid systems - not just MCPs, but cod…

take a peek at https://github.com/willwade/app-automate?tab=readme-ov-file#... - its early and needs some work -but this is the idea behind this.. (my use case is not agents but actual real disabled people..who need tooling to provide better access to the desktop)

Re: Computer Use is 45x more expensive than structured APIs

#225

Post by octoberfranklin »

ls612 wrote: post #217
octoberfranklin wrote: post #206

Earlier quoted context omitted.

How do they know that the "you" accessing the site is the same "you" they previously banned? Face-scanning? Iris patterns?

You used your credit card to buy whatever service or product they sell.

I hate to break it to you but it is really easy to get anonymous visa/mastercard cards.

Re: Computer Use is 45x more expensive than structured APIs

#226

Post by jasomill »

btown wrote: post #143
merlindru wrote: post #44

I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…

If you're on macOS and interested in this space, I highly recommend you open up the system-provided Accessibility Inspector.app and play around with apps and browsers. See how the green cells might guide an LLM to only need to read/OCR specific parts of a screen, how much text is already natively available to the accessibility engine, and how this could lead to really effective hybrid systems - not just MCPs, but cod…

Windows has similar APIs and tools, see, e.g.,

https://accessibilityinsights.io/

https://learn.microsoft.com/en-us/windows/win32/winauto/insp...

https://github.com/FlaUI/FlaUInspect

and for WPF applications specifically,

https://github.com/snoopwpf/snoopwpf

Re: Computer Use is 45x more expensive than structured APIs

#227

Post by jasomill »

merlindru wrote: post #55
gbriel wrote: post #51

Earlier quoted context omitted.

This is a good solution, instead of everyone blowing tokens on repeating the same computer use task, come up with a way to share the workflows. I think you'd need to make sure there aren't workflows shared that extract user information (passwords).

this is protected against at the OS level, provided the applications declare the input correctly as a SecureTextField. i so far haven't found any application that doesn't. all you're able to get out, as far as i can tell, is the length of the entered password.

From applications that capture the screen or use accessibility APIs, perhaps, but what about, e.g., Windows applications that capture window messages, e.g.,

https://devblogs.microsoft.com/cppblog/spy-internals/

Obviously, if you can inject code into a process that receives sensitive data, you're already running in a context where all security bets are off.

But with processes you yourself create, you probably can, even without elevated privileges, unless the application takes measures to prevent injection (akin to game anticheat mechanisms), so it seems worth pointing out that there are simple mechanisms to subvert such "protected" fields that don't require application-specific reverse engineering.

Re: Computer Use is 45x more expensive than structured APIs

#228

Post by oleg2025 »

Couple of months ago I was inspired by kubectl, and built desktopctl CLI to control GUI apps. It uses combination of OCR and Accessibility API on Mac, represents UI as markdown, and exposes actions for mouse and keyboard.

My core idea was that "fast" perception loop is fully local, GPU optimised for UI tokenisation and change detection. "Slow" control loop requires LLM roundtrip, and uses token-efficient markdown interface in CLI output.

It uses relatively stable identifiers for controls, so agents can script common actions, eg `desktopctl pointer click --id btn_save` doesn't require UI tokenisation loop.

https://github.com/yaroshevych/desktopctl/tree/main

Re: Computer Use is 45x more expensive than structured APIs

#229

Post by PetitPrince »

zenoprax wrote: post #204
GarnetFloride wrote: post #175

Earlier quoted context omitted.

My manager just told me that after 12 years of trying to get one of the founders to understand the difference between dev docs and user docs, they tried getting Claude to do it and he finally got it that they are different. He'd been saying this whole time that customer could just read the dev docs. If they could they wouldn't need our software.

How firm is the boundary between a dev doc and a user doc in your opinion? I have found that the overlap can be quite large if the users are also technically proficient. Right now I'm trying to balance "how X works so you can use the app better" with "how X works so you can contribute or build your own plugin". DeepWiki really helps as a backstop for anything not already covered though it's not without its own caveat…

Not OP but I think you have the right intuition in making a difference between using the app / contribute to the app. You may want to read https://diataxis.fr/ which elaborate on this idea and add another dimension (action / cognition) to this.

Re: Computer Use is 45x more expensive than structured APIs

#230

Post by jasomill »

rgilliotte wrote: post #88

[dead]

The nicest thing about this rush to find and build "agentic" endpoints for controlling everything is that there's no reason these same endpoints can't be consumed by deterministic, non-LLM software as well.

It feels like 1994 called, and it's giving me my AppleScript back.

Post reply on HN