Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

tester.army

Post reply on HN
Print view View original

11–20 of 87 posts

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#15

Post by okwasniewski »

Laurel1234 wrote: post #14

Seems interesting, but I wonder about this > Traditional E2E tests are slow to set up and expensive to maintain. Isn't this just using agents to create e2e tests or is there some better new approach I'm missing?

We use agents to navigate the app, making real-time decisions based on its state. I prefer to compare it more to a manual QA engineer than to static e2e tests. We spent a lot of time on the harness to make sure the results are reliable. This allows you to assert on dynamic content like AI-generated content. We also support validation of email flows since the agent can read its own email.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#17

Post by jaggederest »

okwasniewski wrote: post #15
Laurel1234 wrote: post #14

Seems interesting, but I wonder about this > Traditional E2E tests are slow to set up and expensive to maintain. Isn't this just using agents to create e2e tests or is there some better new approach I'm missing?

We use agents to navigate the app, making real-time decisions based on its state. I prefer to compare it more to a manual QA engineer than to static e2e tests. We spent a lot of time on the harness to make sure the results are reliable. This allows you to assert on dynamic content like AI-generated content. We also support validation of email flows since the agent can read its own email.

Fable (rip) is absurdly good at this, great time to build a product around it, you definitely need the harness, but it feels like it just turned the corner to be able to do really in depth and edge case work.

Do you handle heterogenous environments and network connectivity simulation as well? I am working on a mobile app and occasionally having users just lose a request or two can put the state machine into unusual modes.

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#18

Post by poisonborz »

E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?

Re: Launch HN: TesterArmy (YC P26) – Agents that test web and mobile apps

#20

Post by okwasniewski »

jaggederest wrote: post #17
okwasniewski wrote: post #15

Earlier quoted context omitted.

We use agents to navigate the app, making real-time decisions based on its state. I prefer to compare it more to a manual QA engineer than to static e2e tests. We spent a lot of time on the harness to make sure the results are reliable. This allows you to assert on dynamic content like AI-generated content. We also support validation of email flows since the agent can read its own email.

Fable (rip) is absurdly good at this, great time to build a product around it, you definitely need the harness, but it feels like it just turned the corner to be able to do really in depth and edge case work. Do you handle heterogenous environments and network connectivity simulation as well? I am working on a mobile app and occasionally having users just lose a request or two can put the state machine into unusual m…

I feel like new AI model releases will only allow our agents to do more in-depth testing; the space still has a lot of room to grow. Quality assurance is way more complicated than just clicking around a UI.

Regarding the other question: not yet. For now, we have Chromium, iOS, and Android (latest versions of each), but we are working on adding more. Regarding network connectivity, it's coming soon (I have an open PR).

Post reply on HN