Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Agent-desktop – Native desktop automation CLI for AI agents

github.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#41

Post by gvkhna »

tuukkah wrote: post #14
Zetaphor wrote: post #7

Earlier quoted context omitted.

I don't think the accessibility story on Linux is comprehensive enough to make this possible unfortunately. Especially with Wayland. One advantage Mac apps have is they're all targeting the same underlying OS primitives, which is the layer their accessibility platform lives at.

Quote from a sibling comment: - macOS: Accessibility API - Windows: UI Automation - Linux: AT-SPI

Unfortunately AT-SPI support is very hit or miss, mostly miss. To get it useable for this required a huge amount of modification. See my other comment for context.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#42

Post by Zetaphor »

9879875665876 wrote: post #13
Zetaphor wrote: post #7

Earlier quoted context omitted.

I don't think the accessibility story on Linux is comprehensive enough to make this possible unfortunately. Especially with Wayland. One advantage Mac apps have is they're all targeting the same underlying OS primitives, which is the layer their accessibility platform lives at.

There is AT-SPI2: https://invent.kde.org/sdk/selenium-webdriver-at-spi

I'm a heavy Linux user interested in accessibility and I didn't even know this existed.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#44

Post by MattRogish »

TheFragenTaken wrote: post #27

I've long thought about why the tools we have operate on screenshots, and not the accessibility tree. To me the latter would have seemed like the obvious choice from the beginning (structured data), but yet, here we are with pixels. Happy to see progress being made here.

The major limitation is that macOS apps do not have to use the API and so there will always need to be a fallback to something like screen scraping for controls that don’t use it.

Zoom Desktop app is a prime example of this. Many of the windows (join a meeting, settings etc) are normal macOS ones, and those use AX buttons, but many are poorly / weirdly labeled (if at all).

But once the Zoom meeting appears, that’s all (?) custom, and so the best you can do is whatever Zoom decided to offer. The dreaded “this meeting is being recorded” pop up is a custom control and so doesn’t have AX at all; I have automation that basically looks for an appearing window and if it has “OK” just blindly click it and hope for the best.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#48

Post by lahfir »

someone654 wrote: post #17

Looks very interesting. Especially like that language environment is abstracted away, through cli, such that one are not stuck with for example python to write your UI logic (or create your own cli wrapper around PyAutoGUI). How can one help with implementing Linux and Windows support?

Hi, Mac is generally available. Windows and Linux are in the roadmap!

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#49

Post by lahfir »

nerdsniper wrote: post #12
jstanley wrote: post #4

lahfir, I vouched your (currently still dead) comment because it was interesting to me. I expect the reason it is dead is that it seems LLM-generated (you "quietly" launched it on github? Who says that?). Also, your comment claims that the tool is cross-platform and implies that it works on Mac, Windows, and Linux, but the graphic on the github README says it only works on Mac.

It looks hybrid human/LLM at best, but definitely possible that it's mostly human, from someone who is earnestly learning how to use "pitch" language. I got the feeling that some parts, like the bullet points, maybe originated from AI-generated documentation/readme's. My intuition tells me that it could have been AI-generated, but if that's the case then it was heavily edited by a human. I think anyone who went throu…

Hey, thanks for the comment. Yes it's hybrid. AI wrote what I gave as an input. If it's better in articulating a message much better than me, why not use it, right?

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#50

Post by lahfir »

jstanley wrote: post #4

lahfir, I vouched your (currently still dead) comment because it was interesting to me. I expect the reason it is dead is that it seems LLM-generated (you "quietly" launched it on github? Who says that?). Also, your comment claims that the tool is cross-platform and implies that it works on Mac, Windows, and Linux, but the graphic on the github README says it only works on Mac.

Hi. Mac is GA. Windows and Linux are in the roadmap.
Post reply on HN