Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#41

Post by ec109685 »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

There are silly things that trip up e2e tests like a cookie pop up or network failures and whatnot. An AI can plow through these in a way that a purely coded test can’t.

Those types of transient issues aren’t something that you would want to fail a test for given it still would let the human get the job done if it happened in the field.

This seems like the most useful part of adding AI to e2e tests. The world is not deterministic, which AI handles well.

Uber takes this approach here: https://www.uber.com/blog/generative-ai-for-high-quality-mob...

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#42

Post by tomatohs »

worldsayshi wrote: post #6
101008 wrote: post #4

Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.

I would assume that the test runner translates the natural language instruction into a deterministic selector and only re-does that translation when the selector fails. At least that's how I would try to implement it..

This is the right idea and how we do it at TestDriver.ai. The deterministic selector still has about 20% fuzz matching rate, and if it fails it trys to recover.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#43

Post by cschiller »

batikha wrote: post #32

Very cool! I already can see a lot of "this is already solved by playwright/cypress/selenium/deterministic stuff" in the comments. Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a goo…

Thanks for sharing your experience! Completely agree - there's often a huge gap between the perception that testing is "solved" and the reality of manual QA still being necessary, even for core features. We recently had a call with one of the largest US mobile teams and were surprised to learn they're still doing extensive manual testing because some use cases remain uncovered by traditional tools. It's definitely not as "solved" as many might think.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#44

Post by tomatohs »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

You're totally right here, but "debugging failed tests" is a mature problem that assumes you have working tests and people to write them. Most companies don't have the resources to dedicate full engineer time to QA, and if they do nobody maintains the test.

Debugging failed test is a "first world problem"

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#46

Post by ec109685 »

> In terms of trying the product out: since the service is resource-intensive (we provide hosted virtual/real phone instances), we don't currently have a playground available. However, you can see some examples here https://mobileboost.io/showcases and book a demo of GPT Driver testing your app through our website.

Have you considered an approach like what Anthropic is doing for their computer control where an agent runs on your own computer and controls a device simulator?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#47

Post by krainboltgreene »

hn_throwaway_99 wrote: post #27
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

While I agree with your primary pain point, I would argue that that really isn't specific to tests at all. It sounds like what you're really saying is that when something goes wrong, it's really difficult to determine which component in a complex system is responsible. I mean, from what you've described (and from what I've experienced as well), you would have the same if not harder problem if a user experienced a bug…

"OpenTelemetry and other tools are promising, but again, I’ve never seen good enough infra that puts that all together."

It's a two paragraph comment and you somehow missed it.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#48

Post by tomatohs »

ec109685 wrote: post #41
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

There are silly things that trip up e2e tests like a cookie pop up or network failures and whatnot. An AI can plow through these in a way that a purely coded test can’t. Those types of transient issues aren’t something that you would want to fail a test for given it still would let the human get the job done if it happened in the field. This seems like the most useful part of adding AI to e2e tests. The world is not…

I predict an all out war over deterministic vs non-deterministic testing, or at least a new buzzword for fuzzy testing. Product people understand that a cookie banner "shouldn't" prevent the test from passing, but an engineer would entirely disagree (see the rest of the convos below).

Engineers struggle with non-deterministic output. It removes the control and "truth" that engineering is founded upon. It's going to take a lot of work (or again, a toung-in-cheek buzzword like "chaos testing") to get engineers to accept the non-deterministic behavior.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#49

Post by tomatohs »

TechDebtDevin wrote: post #25
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

I doubt that screenshot methods are the bottleneck considering that's the method Microsoft and Anthropic are using.

It's absolutely not the bottleneck. OpenAI can process a full resolution screenshot in about 4 seconds.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#50

Post by tomatohs »

mmaunder wrote: post #35

Congrats! How has Anthropic's latest release supporting computer use affected your planning/thinking around this? PS:If you had this for desktop we'd immediately become a customer.

We do AI E2E desktop, sent you an email.
Post reply on HN