I wonder if we will see people building these kinds of computer use systems to augment traditional QA. What I do not have my mind wrapped around at the moment is how agents might reason across the different states of the interface. It seems like it should be possible to build some kind of state machine for this?
You can't solve computer use by ignoring the interface
31–40 of 42 posts
Re: You can't solve computer use by ignoring the interface
#32You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
You think this blog post is advocating for good UI?
Re: You can't solve computer use by ignoring the interface
#33Earlier quoted context omitted.
Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it.…
> If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is. That's a valid point, yet it's hard to blame authors of OSWorld and ALE. They cre…
Sure, “blame” is a strong word. My point isn’t that anyone deserves actual blame, but just that the models are proving creative. We already know that they will comment out unit tests to make them “pass,” for instance. If we want to focus on a particular skill or behavior, we need the tests to highly constrain the model. If we don’t create our benchmarks like that, shame on us.
Re: You can't solve computer use by ignoring the interface
#34Earlier quoted context omitted.
Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)
Damn what computer are you using, a 2015 Raspberry Pi?
Pinebook. So not much faster. Runs circles around the machines I grew up on.
Re: You can't solve computer use by ignoring the interface
#35Computer use itself is evolving as CLIs and APIs are important again to enable AI use. We'll certainly converge at a happy medium, CLIs, APIs and text for AI, and maybe some form of dynamic interface that the AI builds/adapts on the spot for the exact task you want to execute, on the environment you're at. And of course, good ol' purpose built tools/UIs as we do today, when it makes sense.
Re: You can't solve computer use by ignoring the interface
#36Why is this article based on a chart that has GPT-5.5 and Claude Opus 4.8 as the newest evaluated models? Those are very old now.
There have been massive improvements in computer use in GPT 5.6 and Claude 5.
Re: You can't solve computer use by ignoring the interface
#37Why is this article based on a chart that has GPT-5.5 and Claude Opus 4.8 as the newest evaluated models? Those are very old now. There have been massive improvements in computer use in GPT 5.6 and Claude 5.
Unfortunately, mainly because there're only self-reported and partial numbers for the benchmarks that matter the most.
Re: You can't solve computer use by ignoring the interface
#38Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are s…
Sorry for the confusion, Steelman CTO here. OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions. Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inser…
Thanks, this makes things clearer. Sounds like the idea is that LLMs are competent enough at computer use to bootstrap their way into a terminal or DevTools environment (and from there complete the task using their legitimately impressive scripting skills), but not so good at figuring out how to navigate a UI they've never seen before the way a human does.
Re: You can't solve computer use by ignoring the interface
#39I wonder if we will see people building these kinds of computer use systems to augment traditional QA. What I do not have my mind wrapped around at the moment is how agents might reason across the different states of the interface. It seems like it should be possible to build some kind of state machine for this?
You mean all the possible states of the each element of the interface (buttons, forms, content blocks, etc.)?
Re: You can't solve computer use by ignoring the interface
#40You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.
wow reminds me of the Neopets guild pages