Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Agent-desktop – Native desktop automation CLI for AI agents

github.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#23

Post by Arainach »

tuukkah wrote: post #14
Zetaphor wrote: post #7

Earlier quoted context omitted.

I don't think the accessibility story on Linux is comprehensive enough to make this possible unfortunately. Especially with Wayland. One advantage Mac apps have is they're all targeting the same underlying OS primitives, which is the layer their accessibility platform lives at.

Quote from a sibling comment: - macOS: Accessibility API - Windows: UI Automation - Linux: AT-SPI

The levels of support are radically different. Compositors, window managers, UI frameworks, and apps all have mixed and inconsistent levels of support such that the overall experience is that you simply cannot rely on using a Linux system via accessibility.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#25

Post by ActorNightly »

xnx wrote: post #22

The best desktop automation system would take HDMI input and output USB keystrokes and mouse movements so that it can be plugged into any computer transparently, including work computers.

You don't need hdmi out, just ability to do screenshots, which easy to script.

Arguably though, browser automation gets you 95% of the way there for most things.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#26

Post by xnx »

ActorNightly wrote: post #25
xnx wrote: post #22

The best desktop automation system would take HDMI input and output USB keystrokes and mouse movements so that it can be plugged into any computer transparently, including work computers.

You don't need hdmi out, just ability to do screenshots, which easy to script. Arguably though, browser automation gets you 95% of the way there for most things.

Many systems won't allow the end user to install any software (e.g. work issued laptops), but you can plug in HDMI and USB.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#27

Post by TheFragenTaken »

I've long thought about why the tools we have operate on screenshots, and not the accessibility tree. To me the latter would have seemed like the obvious choice from the beginning (structured data), but yet, here we are with pixels. Happy to see progress being made here.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#28

Post by tidbeck »

TheFragenTaken wrote: post #27

I've long thought about why the tools we have operate on screenshots, and not the accessibility tree. To me the latter would have seemed like the obvious choice from the beginning (structured data), but yet, here we are with pixels. Happy to see progress being made here.

While the accessibility tree is great in many aspects it has its own limitations for example when it comes to stacked views or lazy loading outside the viewport.

Re: Show HN: Agent-desktop – Native desktop automation CLI for AI agents

#30

Post by nlitened »

tidbeck wrote: post #28
TheFragenTaken wrote: post #27

I've long thought about why the tools we have operate on screenshots, and not the accessibility tree. To me the latter would have seemed like the obvious choice from the beginning (structured data), but yet, here we are with pixels. Happy to see progress being made here.

While the accessibility tree is great in many aspects it has its own limitations for example when it comes to stacked views or lazy loading outside the viewport.

I think screenshots also don't help with stacked views and lazy loading outside the viewport
Post reply on HN