Can it farm a ber rune for me?
Yeahh computer-use agents remind me of game automators like RuneScape autoclickers back in the day like SCAR: I posted on this a while back haha https://news.ycombinator.com/item?id=29716900#29720860
Windows-Use: an AI agent that interacts with Windows at GUI layer
21–30 of 30 posts
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#22Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
Agreed. I've noticed ChatGPT when parsing screenshots writes out some Python code to parse it, and at least in the tests I've done (with things like, "what is the RGB value of the bullet points in the list" or similar) it ends up writing and rewriting the script five or so times and then gives up. I haven't tried others so I don't know if their approach is unique or not, but it definitely feels really fragile and slo…
I noticed something similar. I asked it extract a guid from an image and it wrote a python script to run ocr against it...and got it wrong. Prompting a bit more seemed to finally trigger it to use it's native image analysis but I'm not sure what the trick was.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#23Cool. Reminds me of using SendKeys() in Visual Basic 6 in the 90s https://learn.microsoft.com/en-us/dotnet/api/microsoft.visua...
SendKeys() in VB powered basically all of the AOL chat bots in the 90’s.
It’s how I accidentally learned the Win32 API
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#24Earlier quoted context omitted.
Agreed. I've noticed ChatGPT when parsing screenshots writes out some Python code to parse it, and at least in the tests I've done (with things like, "what is the RGB value of the bullet points in the list" or similar) it ends up writing and rewriting the script five or so times and then gives up. I haven't tried others so I don't know if their approach is unique or not, but it definitely feels really fragile and slo…
I noticed something similar. I asked it extract a guid from an image and it wrote a python script to run ocr against it...and got it wrong. Prompting a bit more seemed to finally trigger it to use it's native image analysis but I'm not sure what the trick was.
I've run into this with uploading audio and text files, have to yell at it to not write any code and use it's native abilities to do the job.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#25genuinely asking, what do you think are the use cases for someone requiring this?
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#26Cool. Reminds me of using SendKeys() in Visual Basic 6 in the 90s https://learn.microsoft.com/en-us/dotnet/api/microsoft.visua...
SendKeys() in VB powered basically all of the AOL chat bots in the 90’s. It’s how I accidentally learned the Win32 API
Me too! With Sendkeys and some Win32 API calls, I wrote an AOL add-on (available through Keyword: addons) called AoLOL!. It was my first software business.
Q: How do you identify the AOL window? A: Look for an app with titlebar = "America[space][space]Online"
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#27Awesome job! I'm working on a similar Agent that's highly dependent on AutoIt.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#28Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
Important is subjective — In the healthcare space, I’d make the claim that most applications don’t expose themselves correctly (native or web). CV and direct mouse/kb interactions are the “base” interface, so if you solve this problem, you unlock just about every automation usecase. (I agree that if you can get good, unambiguous, actionable context from accessibility/automation trees, that’s going to be superior)
I’ve been working hard on our new component implementation (Vue/TS) to include accessibility for components that aren’t just native reskins, like combo and list boxes, and keyboard interactivity is a real pain. One of my engineers had it half-working on her dropdown and threw in the towel for MVP because there’s a lot of little state edge cases to watch out for.
Thankfully the spec as provided by MDN for minimal functionality is well spelled out and our company values meeting accessibility requirements, so we will revisit and flesh out what we’re missing.
Also I wanna give props (ha) to the Storybook team for bringing accessibility testing into their ecosystem as it really does help to have something checking against our implementations.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#29Earlier quoted context omitted.
Agreed. I've noticed ChatGPT when parsing screenshots writes out some Python code to parse it, and at least in the tests I've done (with things like, "what is the RGB value of the bullet points in the list" or similar) it ends up writing and rewriting the script five or so times and then gives up. I haven't tried others so I don't know if their approach is unique or not, but it definitely feels really fragile and slo…
I noticed something similar. I asked it extract a guid from an image and it wrote a python script to run ocr against it...and got it wrong. Prompting a bit more seemed to finally trigger it to use it's native image analysis but I'm not sure what the trick was.
Probably just ask it to use native image analysis versus writing code. I have done this before extracting usernames from screenshots.
Re: Windows-Use: an AI agent that interacts with Windows at GUI layer
#30Very cool - does anyone know of an OSX equivalent? Preferably one that is similarly able to understand and interact with web page elements, in addition to app elements and system elements.