[flagged]
You can't solve computer use by ignoring the interface
11–20 of 42 posts
Re: You can't solve computer use by ignoring the interface
#12You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
[deleted]
Re: You can't solve computer use by ignoring the interface
#13You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)
Re: You can't solve computer use by ignoring the interface
#14You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
[flagged]
Re: You can't solve computer use by ignoring the interface
#15We have 2 paths here: - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates. - or make AI systems use the screen like humans do - what the article is arguing for Both of these paths are possible. One of them is much more efficient than the other in the long…
> - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
Yea, hasn't worked so far, devs and orgs have been pushing for APIs ages, same with structured data, eg schema.org
The third option is agent to agent communication, mcp, etc.
Re: You can't solve computer use by ignoring the interface
#16Is no benchmark properly sandboxed? It feels like every single the logs are provided for a benchmark run the LLM is cheating in a way that should have been clearly blocked by a sandbox or the harness.
Re: You can't solve computer use by ignoring the interface
#17Looking how much agents like to use and push to have functional a11y trees, we may accidentally solve accessibility as well
Haha, one day, one day...
Re: You can't solve computer use by ignoring the interface
#18Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.
Re: You can't solve computer use by ignoring the interface
#19You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
The words "trail on mouse" instantly reminded me of the Windows mouse trails, that's what I was expecting going into the article.
But honestly, I really liked whatever you call this type of trail. Not sure this was the intent, but I used the trail to trace under the lines as I reading them which helped me pay attention to the text a bit better.
100% agree with you on messing with selection and copy/paste though.
Re: You can't solve computer use by ignoring the interface
#20The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sold by the company. At some level, I don’t fault the AI for taking the API path if it’s available. In fact, I’m impressed that it found the API and used it correctly. That doesn’t seem like an argument that the sky is falling.