Why using Claude Sonnet for navigating the gui? You could also use selenium, playwright or some other headless browser instead. Would be much cheaper and more reproducible.
Computer Use is 45x more expensive than structured APIs
271–280 of 295 posts
Re: Computer Use is 45x more expensive than structured APIs
#272This aligns with what I've seen in production. Forcing an LLM to navigate a DOM is an incredible waste of compute when an API is available.
Re: Computer Use is 45x more expensive than structured APIs
#273Earlier quoted context omitted.
It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perceive…
My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything" I guess that just never occurred to anybody before.
We used to document things. Apple had a series of beautiful books documenting ‘Inside Macintosh’. Visual C++ used to come with 14 inches of decent books.
Then books gave way to the web and there was no longer a publishing deadline or copy editor and we just kinda stopped caring. The users still cared, but many producers stopped.
We had an art and we gave it up. Not completely, but substantially.
Re: Computer Use is 45x more expensive than structured APIs
#274I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…
If agents is what it finally takes to get good a11y I'll take it. I'll bitch about it, but I'll take it.
as someone who doesnt do web stuff, i found some humor in having no idea what "a11y" was, having to look it up, and finding out it is supposed to be "accessibility".
my quick accessibility tip: introduce what your acronyms, initialisms, and numeronyms stand for at least once.
Re: Computer Use is 45x more expensive than structured APIs
#275Earlier quoted context omitted.
It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perceive…
> It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. > It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perc…
The problem with up front spec writing was that it created this long, very high risk period where there was no code, no prototype, no hard limits.
So you're arguing back and forth about exactly which fields should be collected where and meanwhile the world is changing around you. The person who insisted on an absolutely minimal signup page has changed jobs, the new hire prioritizes complete data even if it means more signup bounces.
Weeks and months are going by and there's nothing to click on and still nothing to tether the project to reality or limit the scope of debate. High functioning teams can avoid this, but they don't always control who inserts themselves into the conversation.
And of course as soon as coding starts the spec is out of date unless you absolutely nailed it which I've never seen happen in 15 years. Once a user sees the software and gives feedback it probably needs to be rewritten. Maintaining the spec (pre-LLM) was not that much less work than maintaining the software itself. All this time the world around the software is changing and the spec needs to be updated to reflect that.
And while the spec helps in understanding the system, ultimately you still need to understand the code and it.
But now the time between spec and working code is greatly reduced and spec updating can be automated to an extent. The cost is greatly reduced and the benefits have increased, so people like specs now.
Re: Computer Use is 45x more expensive than structured APIs
#276Earlier quoted context omitted.
The O’Reilly animal for Jira is apparently some kind of duck or goose. Matthew B. Doar (2011). Practical JIRA Plugins . O’Reilly. https://www.oreilly.com/library/view/practical-jira-plugins/... In case anyone was wondering. Which they probably weren’t :p
I'm more interested in the next volume: impractical Jira plugins
I'm sure the doohickeycorporation folks on Reddit can come up with some.
Re: Computer Use is 45x more expensive than structured APIs
#277Earlier quoted context omitted.
My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything" I guess that just never occurred to anybody before.
The CEO of Uber made the same comment on Diary of a CEO recently. I think it was for their customer service team if I'm not mistaken, they threw their existing docs at an LLM and it was all over the place because policies were poorly documented and defined. The team is now documenting everything from scratch, focusing on outcomes rather than process - TBD if it works out.
which isnt a horrible overall result for customers, other than the gap time before the good and useable documentation is put together.
Re: Computer Use is 45x more expensive than structured APIs
#278Earlier quoted context omitted.
> It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. > It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perc…
The problem with up front spec writing was that it created this long, very high risk period where there was no code, no prototype, no hard limits. So you're arguing back and forth about exactly which fields should be collected where and meanwhile the world is changing around you. The person who insisted on an absolutely minimal signup page has changed jobs, the new hire prioritizes complete data even if it means more…
What I hated is when you come into an old project. The spec says one thing and the code contradicts it. WTF do you do? I felt more like an archeologist than a programmer.
Re: Computer Use is 45x more expensive than structured APIs
#279Earlier quoted context omitted.
The CEO of Uber made the same comment on Diary of a CEO recently. I think it was for their customer service team if I'm not mistaken, they threw their existing docs at an LLM and it was all over the place because policies were poorly documented and defined. The team is now documenting everything from scratch, focusing on outcomes rather than process - TBD if it works out.
which isnt a horrible overall result for customers, other than the gap time before the good and useable documentation is put together.
I'm in favor of better documentation generally, and if LLM use does remain pervasive in software I'm sure we'll all live in a new normal where most of the job is documentation of the product and test scenarios. If they ever become cheap enough, I'd expect it to be pretty common to run smoke testing of sorts, pointing the LLM at specs and having it click around to look for discrepancies.
Re: Computer Use is 45x more expensive than structured APIs
#280Earlier quoted context omitted.
"In an agentic world, the OS needs to be completely rethought" - if AI is progressing as fast as we think it is, I don't think we'll be interested in waiting for the world to rebuild all the legacy tooling from the OS up. For new stuff, that'd be great. I imagine the AIs will get a lot better at intercepting things at an intermediate level - API calls under the hood, etc. Probably much better (and cheaper) vision abi…
Maybe not the same approach, but start with a kernel that acts as a governed membrane for everything else? Open source research/project I have been exploring on the topic: https://aevum.build/learn/architecture/
Worse is better. My bet is we'll still be running primarily Linux based servers 25 years from now.