Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can't solve computer use by ignoring the interface

steelmanlabs.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: You can't solve computer use by ignoring the interface

#21

Post by euphetar »

ameliaquining wrote: post #18

Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are s…

Sorry for the confusion, Steelman CTO here.

OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.

Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.

WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.

Re: You can't solve computer use by ignoring the interface

#22

Post by euphetar »

drob518 wrote: post #20

The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sol…

I get the point, but you can't always use the API.

1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task 2. Sometimes it just doesn't work.

https://osworld-v2-monitor.xlang.ai/task/tasks/068

Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task.

The funniest part is that it needs to achieve a score of 100, but puts 150 "just in case". I don't think I want it to take the same approach when e.g. fixing a customer's balance.

Re: You can't solve computer use by ignoring the interface

#23

Post by euphetar »

kvark wrote: post #9

We have 2 paths here: - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates. - or make AI systems use the screen like humans do - what the article is arguing for Both of these paths are possible. One of them is much more efficient than the other in the long…

Honestly I think we will get something in the middle. A lot of stuff will become API calls, but we will also have a lot of people-facing interfaces, and the need to use them automatically (for QA at the very least least). And then there are games

Re: You can't solve computer use by ignoring the interface

#24

Post by drob518 »

euphetar wrote: post #22
drob518 wrote: post #20

The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sol…

I get the point, but you can't always use the API. 1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task 2. Sometimes it just doesn't work. https://osworld-v2-monitor.xlang.ai/task/tasks/068 Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task. The funniest part is that it n…

Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it. If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.

Re: You can't solve computer use by ignoring the interface

#25

Post by mpavlov »

charcircuit wrote: post #16

Is no benchmark properly sandboxed? It feels like every single the logs are provided for a benchmark run the LLM is cheating in a way that should have been clearly blocked by a sandbox or the harness.

There's an anecdotal paper 'How We Broke Top AI Agent Benchmarks: And What Comes Next' https://moogician.github.io/blog/2026/trustworthy-benchmarks...

Re: You can't solve computer use by ignoring the interface

#26

Post by twright »

winddude wrote: post #15
kvark wrote: post #9

We have 2 paths here: - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates. - or make AI systems use the screen like humans do - what the article is arguing for Both of these paths are possible. One of them is much more efficient than the other in the long…

> - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates. Yea, hasn't worked so far, devs and orgs have been pushing for APIs ages, same with structured data, eg schema.org The third option is agent to agent communication, mcp, etc.

Throwback to a brief moment when everyone wanted to support IFTTT. If I remember, it ran out of steam because it takes away eyeballs from the actual application (social networks in-particular). The use cases are also rather niche.

Re: You can't solve computer use by ignoring the interface

#28

Post by mpavlov »

drob518 wrote: post #24
euphetar wrote: post #22

Earlier quoted context omitted.

I get the point, but you can't always use the API. 1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task 2. Sometimes it just doesn't work. https://osworld-v2-monitor.xlang.ai/task/tasks/068 Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task. The funniest part is that it n…

Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it.…

> If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.

That's a valid point, yet it's hard to blame authors of OSWorld and ALE. They created an env for benchmarking long horizon task completion to be as close to real computer as possible. And for this goal CLI/API access is generally useful, yet when the model not defaults to it for the majority of subtasks.

There're benchmarks that would measure UI literacy (Webgames Benchmark is one). But they are far from the task we want to benchmark in the end.

Re: You can't solve computer use by ignoring the interface

#29

Post by dartharva »

bluGill wrote: post #13
juancn wrote: post #6

You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.

Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)

Damn what computer are you using, a 2015 Raspberry Pi?
Post reply on HN