Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Deltix – AI Driven Testing

app.deltix.ai

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Show HN: Deltix – AI Driven Testing

#11

Post by _puk »

jeffnash wrote: post #8
iamcoder18 wrote: post #3

Just did this locally by giving my AI agent full ADB access to a test phone I have. It's actually kinda amazing just watching it swipe, open the app launcher, and find issues to fix later.

What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator). I wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a dae…

By coordinate/ UIAutomator I'm assuming you mean the likes of scrcpy[0]?

Would be great if you have a way to do it programmatically.

https://github.com/Genymobile/scrcpy

Re: Show HN: Deltix – AI Driven Testing

#12

Post by jeffnash »

_puk wrote: post #11
jeffnash wrote: post #8

Earlier quoted context omitted.

What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator). I wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a dae…

By coordinate/ UIAutomator I'm assuming you mean the likes of scrcpy[0]? Would be great if you have a way to do it programmatically. https://github.com/Genymobile/scrcpy

I didn't end up needing to go full scrcpy because my use case just runs Android in an emulator (Waydroid) for my Hermes agent to run apps. If I need vision or (x,y) coordinate interaction, I can just screenshot the emulator window and interact via ADB. However, I ended up using the accessibility tree to create a semantic, ephemeral ref API as the preferred mechanism for interaction. Basically more 'Playwright for Android' than 'Remote Desktop for Android'. But, unfortunately, the accessibility tree is often incomplete or inaccurate.

https://www.djfa.ai/articles/clawdroid

Re: Show HN: Deltix – AI Driven Testing

#14

Post by iamcoder18 »

jeffnash wrote: post #8
iamcoder18 wrote: post #3

Just did this locally by giving my AI agent full ADB access to a test phone I have. It's actually kinda amazing just watching it swipe, open the app launcher, and find issues to fix later.

What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator). I wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a dae…

Mine just took screenshots but I had wanted to set it up with accessibility too. That's a cool way to do that.
Post reply on HN