I feel vaguely vindicated that the agent can't figure out how to use the modern Save as workflow, either, and reverts to the traditional dialog.
Windows-Use: an AI agent that interacts with Windows at GUI layer
11–20 of 30 posts
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#12Cool. Reminds me of using SendKeys() in Visual Basic 6 in the 90s https://learn.microsoft.com/en-us/dotnet/api/microsoft.visua...
And BeOS/Haiku with the "Hey" command which does literally the same, but far more than key input. You can interact with widgets too.
Under Unix, there's xdotool and friends.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#13Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
Important is subjective — In the healthcare space, I’d make the claim that most applications don’t expose themselves correctly (native or web).
CV and direct mouse/kb interactions are the “base” interface, so if you solve this problem, you unlock just about every automation usecase.
(I agree that if you can get good, unambiguous, actionable context from accessibility/automation trees, that’s going to be superior)
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#14Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
I recently tried using Qwen VL or Moondream to see if off-the-shelf they would be able to accurately detect most of the interesting UI elements on the screen, either in the browser or your average desktop app.
It was a somewhat naive attempt, but it didn't look like they performed well without perhaps much additional work. I wonder if there are models that do much better, maybe whatever OpenAI uses internally for operator, but I'm not clear how bulletproof that one is either.
These models weren't trained specifically for UI object detection and grounding, so, it's plausible that if they were trained on just UI long enough, they would actually be quite good. Curious if others have insight into this.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#15Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
Most Electron software doesn't follow accessibility guidelines and exposes nothing over UIA
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#16Cool. Reminds me of using SendKeys() in Visual Basic 6 in the 90s https://learn.microsoft.com/en-us/dotnet/api/microsoft.visua...
I loved SendKeys()!
Used it to write programs that would run in the background & spook my friends by "typing" quotes from movies at random times on their computer.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#17Working on something very similar in Rust. It's quite magical when it works (that's a big caveat, as I'm trying to make it work with local LLMs). Very cool implementation, and imo, this is the future of computing.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#18Can it farm a ber rune for me?
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#19Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
Important is subjective — In the healthcare space, I’d make the claim that most applications don’t expose themselves correctly (native or web). CV and direct mouse/kb interactions are the “base” interface, so if you solve this problem, you unlock just about every automation usecase. (I agree that if you can get good, unambiguous, actionable context from accessibility/automation trees, that’s going to be superior)
[deleted]
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#20I remember an older friend asking me recently; will there be a thing soon where I can make my computer go on auto-pilot?
I guess I can answer, "yes I think so."