Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

tester.army

Post reply on HN
Print view View original

21–30 of 87 posts

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#21

Post by okwasniewski »

poisonborz wrote: post #18

E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e t…

Unfortunately from our experience tests don’t scale as well as code. First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots.

To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.

Regarding test case management, our customers have used our CLI to migrate their existing test cases from whatever system they were using before.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#23

Post by okwasniewski »

tcoff91 wrote: post #22

I'm curious how your mobile testing compares to https://revyl.com I've been experimenting with Revyl and it's really nice. I think this agent-driven testing is the future.

We support both web and mobile, which is what a lot of companies prefer, just one agent for both. Also, I'm pretty sure Revyl relies only on vision models, which tend to be slower. We built the platform around a hybrid approach that combines vision and accessibility APIs, which is much faster.

Would love to hear your feedback after you try it out!

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#24

Post by thih9 »

okwasniewski wrote: post #10
iknownthing wrote: post #8

.army?

We are thinking whether to change this.. We also have testerarmy.com/.ai

Change it now to .com or get stuck there for years, suffering anti spam filters, potential renewal problems and more in the meantime.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#28

Post by tootubular »

okwasniewski wrote: post #10
iknownthing wrote: post #8

.army?

We are thinking whether to change this.. We also have testerarmy.com/.ai

You have the .com? That's a no-brainer imo. I have a domain for a saas where the .com is squatted so I settled for .ai (and other surrounding TLDs / host permutations) and right out of the gate ran into some issues with firewall vendors in corpo environments.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#29

Post by okwasniewski »

j0sip wrote: post #26

I wonder how does it compare to mobileboost.io, which has been used by some companies like Duolingo?

Our approach is heavily focused on agents, both for executing tests and for managing the platform. We want to provide the best and simplest way to conduct agentic testing, with a strong focus on details. It looks like their platform also requires a sales call.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#30

Post by okwasniewski »

zuzululu wrote: post #25

not sure the pain point you mentioned resonate. with LLMs its very easy to do E2E testing. also I feel uneasy about outsourcing this part with all the security issues these days.

Unfortunately from our experience tests don’t scale as well as code.

First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox), spinning up simulators and handling video recording and screenshots.

To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.

Regarding security part, the product can operate solely without any access to the codebase, you can just give us a URL or a mobile app build and we will do the testing.

Post reply on HN