Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drive any macOS app in the background without stealing the cursor

github.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#21

Post by jonhohle »

jorvi wrote: post #5
LatencyKills wrote: post #2

Ex-Apple engineer here. I really like your implementation. A few years ago I built a similar tool to help me automate the testing of some of my native macOS apps. Being able to run multiple UI automation tests simultaneously was the big win in my case. My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

If Charmin put sensors in toilet paper rolls to optimize the wiping experience, it would be dystopian. Why do we give software a pass? Privacy is a right not a telemetry problem and opt-out by default is non-consensual surveillance.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#22

Post by frabonacci »

j-conn wrote: post #20

Incredible! I’m interested in doing something similar on windows, have you looked into that at all? Apparently codex computer use plans to support this on windows in the future. Were you able to see how codex was doing it, or the inspiration was just “they’ve shown it’s possible”?

Thanks! We haven't gone deep on Windows yet because we're still focused on polishing the macOS release. We want to go deeper on the Mac experience before going broader across platforms, and there are still a lot of features we want to ship and use cases we want to share.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#23

Post by frabonacci »

pimlottc wrote: post #19

What is specific about this for using with agents? As opposed to offering it as a general automation library for any use?

Nothing prevents using it as a general automation library.

If you want to use it directly as an automation framework, you can take a Swift dependency on 'CuaDriverCore': https://cua.ai/docs/cua-driver/guide/getting-started/swift-i...

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#25

Post by dewey »

pnw_throwaway wrote: post #12
jorvi wrote: post #5

Earlier quoted context omitted.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

The problem with opt-in telemetry is that 95% of users are sick and tired of being spied on with every little thing they do.

As you can see with TikTok / Instagram usage…regular people who are not on HN could not care less about that.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#26

Post by eddyg »

pnw_throwaway wrote: post #12
jorvi wrote: post #5

Earlier quoted context omitted.

The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfoodi…

The problem with opt-in telemetry is that 95% of users are sick and tired of being spied on with every little thing they do.

Telemetry (if it’s truly telemetry) is nowhere close to “tracking”. People conflate the two all the time. One can provide useful, anonymous metrics (e.g. “user enabled feature X”) without doing anything but incrementing the counter for “feature X”.

The “Firefox Problem” is that all the power users disable telemetry, so all the “cool” features that power users like (but never get used by “regular people”) get ignored or removed instead of improved because, according to the metrics, “nobody uses them”.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#28

Post by frabonacci »

BenFranklin100 wrote: post #27

Being new to the idea of using agents to run programs on one’s computer, could someone provide several use cases?

A few examples i'm excited about:

- Closing the coding feedback loop by having agents verify their own changes in a real app

- Automating repetitive workflows across apps that don't have good APIs

- Agents recording product demos of them using software. One compelling use case here: https://x.com/trycua/status/2047383207612645426

- Creating CLI and APIs for apps by reverse implementing their GUI, e.g. see: https://github.com/HKUDS/CLI-Anything

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#29

Post by prashant3210 »

alsetmusic wrote: post #9

I tried out their Loom vm software a couple of months back. Worked well, fwiw. I'm not using it anymore because I decided to just give agents direct (supervised) access to my devices.

Same here. I give agents supervised direct access on my Mac for a side project. Session stealing is annoying. VM feels overkill for solo dev, but hate that the cursor jumps around while I try to do other things. Background driver sounds like the missing middle ground.

Re: Show HN: Drive any macOS app in the background without stealing the cursor

#30

Post by fragmede »

prashant3210 wrote: post #29
alsetmusic wrote: post #9

I tried out their Loom vm software a couple of months back. Worked well, fwiw. I'm not using it anymore because I decided to just give agents direct (supervised) access to my devices.

Same here. I give agents supervised direct access on my Mac for a side project. Session stealing is annoying. VM feels overkill for solo dev, but hate that the cursor jumps around while I try to do other things. Background driver sounds like the missing middle ground.

http://tart.run makes the VM part easy. So what if it's overkill?
Post reply on HN