Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drive any macOS app in the background without stealing the cursor

github.com

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#11

Post by frabonacci »

kveykva wrote: post #4
LatencyKills wrote: post #2

Ex-Apple engineer here. I really like your implementation. A few years ago I built a similar tool to help me automate the testing of some of my native macOS apps. Being able to run multiple UI automation tests simultaneously was the big win in my case. My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.

Would you be open to sharing what you built for running the automation tests? I could really use this right now.

We don't have a specific testing framework yet. cua-driver is closer to an automation interface than a test runner. that said, you could definitely build one on top of it. For reference these are some of our integration tests: https://github.com/trycua/cua/tree/main/libs/cua-driver/Test...

One useful trick is to cua-driver 'launch_app' instead of the default 'open' or other osascript, since it can start the app without raising/focusing it, and the tests don't disturb your active desktop while they run

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#12

Post by pnw_throwaway »

jorvi wrote: post #5
LatencyKills wrote: post #2

Ex-Apple engineer here. I really like your implementation. A few years ago I built a similar tool to help me automate the testing of some of my native macOS apps. Being able to run multiple UI automation tests simultaneously was the big win in my case. My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

The problem with opt-in telemetry is that 95% of users are sick and tired of being spied on with every little thing they do.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#13

Post by frabonacci »

alsetmusic wrote: post #9

I tried out their Loom vm software a couple of months back. Worked well, fwiw. I'm not using it anymore because I decided to just give agents direct (supervised) access to my devices.

Thanks for trying out Lume! We definitely haven't given up on the idea of sandboxing GUI agents in local macOS VMs. Cua Driver is aimed at a different use case though, letting coding agents and general agents use the Mac you're already on, asynchronously and in the background. That also makes the economics better since multiple agents can share the same machine instead of each needing its own VM

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#14

Post by dtran »

This is one of the coolest hacks I've seen recently. Having done some much less involved MacOS hacking, I can't help but wonder if we may finally see momentum behind some flavor of agent-friendly Linux/Android if Apple doesn't give us more ways to let agents interact with our machines.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#15

Post by frabonacci »

dtran wrote: post #14

This is one of the coolest hacks I've seen recently. Having done some much less involved MacOS hacking, I can't help but wonder if we may finally see momentum behind some flavor of agent-friendly Linux/Android if Apple doesn't give us more ways to let agents interact with our machines.

really appreciate it. macOS has powerful primitives already, but they weren’t designed as one coherent agent API so you end up stitching together and hitting roadblocks. If Apple doesn't make this more first-class, Linux/Android-style environments may move faster because they’re easier to instrument. I think the OpenAI/Jony Ive AI hardware rumors are yet another signal that people may start building agent-native CUA devices instead of retrofitting agents onto existing desktops

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#16

Post by jorvi »

pnw_throwaway wrote: post #12
jorvi wrote: post #5

Earlier quoted context omitted.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

The problem with opt-in telemetry is that 95% of users are sick and tired of being spied on with every little thing they do.

If they really were they would turn it off. And stop using Gmail and Android.

The overwhelming majority of people don't care about digital privacy because the cost is opaque to them.

Also, telemetry when done right isn't "spying". Again, it is anonymized and used to see, for example, where the hot paths and paper cuts in applications are.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#17

Post by jorvi »

crazygringo wrote: post #10
jorvi wrote: post #5

Earlier quoted context omitted.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

I'm confused. You claim power users opt in to telemetry, and then immediately say power users opt out.

A subset of power users want to their usage to be profiled (me, if I trust the company. Brave, Mozilla, Mullvad, 1Password, Bitwarden, Valve, companies like that). But most power users will not want that because of privacy worries.

From that you get two situations.

Opt-in:

- Regular users: click all 'ok' through setup at lightning speed, no telemetry enabled.

- Most power users: consciously don't check the box to opt-in because of privacy worries.

- Big picture power users: consciously check the opt-in box given they trust you (because they want their usage patterns to be profiled and optimized for).

Opt-out:

- Regular users: click all 'ok' through setup at lightning speed, telemetry enabled.

- Most power users: consciously check the box to opt-out because of privacy worries.

- Big picture power users: consciously don't check the opt-out box given they trust you (because they want their usage patterns to be profiled and optimized for).

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#18

Post by awwaiid »

crazygringo wrote: post #10
jorvi wrote: post #5

Earlier quoted context omitted.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

I'm confused. You claim power users opt in to telemetry, and then immediately say power users opt out.

power users opt in to opt in telemetry, and power users opt out of opt out telemetry. Power users click all the buttons.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#20

Post by j-conn »

Incredible! I’m interested in doing something similar on windows, have you looked into that at all? Apparently codex computer use plans to support this on windows in the future. Were you able to see how codex was doing it, or the inspiration was just “they’ve shown it’s possible”?
Post reply on HN