[flagged]
Computer Use is 45x more expensive than structured APIs
231–240 of 295 posts
Re: Computer Use is 45x more expensive than structured APIs
#232Couple of months ago I was inspired by kubectl, and built desktopctl CLI to control GUI apps. It uses combination of OCR and Accessibility API on Mac, represents UI as markdown, and exposes actions for mouse and keyboard. My core idea was that "fast" perception loop is fully local, GPU optimised for UI tokenisation and change detection. "Slow" control loop requires LLM roundtrip, and uses token-efficient markdown int…
I've learned that compared to APIs, human interfaces are slow and messy, but there is actually a lot of science behind them. The good apps expose information well, and are optimised for clicks, typing, etc.
The best GUIs make great use of muscle memory, which makes them perfect candidates for scripting via CLI. eg a simple sequence "open Notes app, hit Cmd+F, enter search term, read list of results" can be one Bash command invoked by AI agent.
Re: Computer Use is 45x more expensive than structured APIs
#233Earlier quoted context omitted.
It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perceive…
> It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. > It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perc…
Yeah I think a lot of pushback to best practices is basic cost/benefit; I like writing documentation, but I'm also often feeling a bit depressed that nobody will actually read it in as much detail as I wrote it. But LLMs do / can.
Actually there's a lot of projection there too; I don't read documentation in detail. And nowadays, I point an LLM at documentation so that it can find the details I would otherwise skip over.
The destruction of the millennial attention span is real, and it's worse in the younger generations, lmao.
Re: Computer Use is 45x more expensive than structured APIs
#234Earlier quoted context omitted.
My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything" I guess that just never occurred to anybody before.
AI might actually RTFM
It would / should / can, but there's big efforts in reducing token consumption now, so AI will likely try to skim and pick documentation just like real humans.
Re: Computer Use is 45x more expensive than structured APIs
#235Earlier quoted context omitted.
You used your credit card to buy whatever service or product they sell.
I hate to break it to you but it is really easy to get anonymous visa/mastercard cards.
And easy to identify anonymous cards.
https://www.google.com/search?q=identify+anonymous+visa+mast...
Re: Computer Use is 45x more expensive than structured APIs
#236Earlier quoted context omitted.
Why on earth would they offer convenient hooks for AI chatbots? Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more. Watch how fast Meta adds this if a new hot shot social media app succeeds by designing for AI agents controlled by users.
>Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more. This is the exact opposite of what will happen (and in fact what has happened). Reddit is suing Perplexity right now for scraping. Meta will not serve content to some other app for free - for what benefit? They will not see advertisi…
Scraping and asking an agent is different.
Re: Computer Use is 45x more expensive than structured APIs
#237Earlier quoted context omitted.
> It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. > It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perc…
Yeah I think a lot of pushback to best practices is basic cost/benefit; I like writing documentation, but I'm also often feeling a bit depressed that nobody will actually read it in as much detail as I wrote it. But LLMs do / can. Actually there's a lot of projection there too; I don't read documentation in detail. And nowadays, I point an LLM at documentation so that it can find the details I would otherwise skip ov…
Well it's also just that you have a list of 20 features to add, and if it works, you want to ship it, and someone might even get mad if you spend a day dawdling on best practices and documentation and so on. Corporate cultures generally don't have the same long term thinking about reusability and legibility and fault-tolerance that an individual coder may have about the code they want to write once and forget. (Neither do LLMs, for that matter).
Re: Computer Use is 45x more expensive than structured APIs
#238Earlier quoted context omitted.
My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything" I guess that just never occurred to anybody before.
The CEO of Uber made the same comment on Diary of a CEO recently. I think it was for their customer service team if I'm not mistaken, they threw their existing docs at an LLM and it was all over the place because policies were poorly documented and defined. The team is now documenting everything from scratch, focusing on outcomes rather than process - TBD if it works out.
Yeah, someone made the point in a popular post here recently that all the firings are reducing institutional knowledge. IMHO, replacing that knowledge with LLM-written documentation is even more potentially catastrophic. Just from organizations I've worked in, a lot of the useful human knowledge is in knowing how to handle either undocumented edge cases or situations where the documents are outdated or wrong. Working with LLMs and reminding them to update those docs every time? Good luck. And if it's something where the docs touch actual real world operations, that's an area where only human operators with hands-on experience are going to recognize the potential conflicts or cognitive dissonance.
Re: Computer Use is 45x more expensive than structured APIs
#239[flagged]
Re: Computer Use is 45x more expensive than structured APIs
#240I want to just talk to the Mac and have it do things. I tried computer use and other alternatives, but the latency made it unusable. I want to be able to control both Mac, apps and the browser. I also need it to figure out things by itself given a goal. Claude Code with the --chrome flag is kind of good, but it's too slow. I wanted to try faster APIs, like the one hosted on Cerebras, but it's too expensive. Any solut…
Do you want to do something that can't be done through AppleScript, macOS accessibility APIs, and something like Puppeteer to control the browser?
Or something you don't understand how to do manually?
Because I guess I don't understand the attraction of using an LLM for system automation where existing interfaces exist, other than as a form of documentation, or to write code using these interfaces.