Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#11

Post by cschiller »

codepathfinder wrote: post #3

Is it possible to record the user screen and just generate a test case. I believe that's most efficient way IMO

Yes, great point! We have an 'Assistant' feature where you can perform the flow on the device, and we automatically generate the test case as you navigate the app. As you mentioned, it’s a great starting point to quickly automate the functional flow. Afterwards, you can add more detailed assertions as needed. Technically we do this by using both the UI hierarchy from the app as well as vision models to generate the test prompt.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#12

Post by chairhairair »

joshuanapoli wrote: post #10
101008 wrote: post #4

Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.

I think that the hope/dream here is to make end-to-end tests less flakey. It would be great to have navigation and assertions commands that are robust against simple changes in the app that aren't relevant to the test case.

It's just a dream then.

It's completely at-odds with the strengths of LLMs (fuzzy associations, rough summaries, naive co-thinking).

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#13

Post by cschiller »

codepathfinder wrote: post #2

I've been a mobile developer for the past 10 years and my overall belief is that mobile app development has slower growth and companies with the mobile team are investing less on mobile Dev or testing+tooling+education. Do you think the market is still hot once it was to use your product?

I would say that mobile apps are still the primary format for launching new consumer services, incl. new apps like ChatGPT and many others. However we’ve observed that teams are expected to do more with less—delivering high-quality products while ensuring compliance, often with the same or even smaller team sizes. This is why we focus on minimizing the engineering burden, particularly when it comes to repetitive tasks like regression testing, which can be especially painful to maintain in the mobile ecosystem due to use of third-party integrations (authentication, payments, etc.).

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#14

Post by digging »

aksophist wrote: post #7
101008 wrote: post #4

Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.

It’s only deterministic for each version of the app. Versions change: UI elements move, change their title slightly. Irrelevant promo popups appear, etc. For a deterministic solution, someone has to go and update the tests to handle all of that. Good ‘accessibility hygiene’ can help, but many apps lack that. And then there are truly dynamic apps like games or simulators. There may be no accessibility info to determin…

> Versions change: UI elements move, change their title slightly

Not randomly, I'd hope. I think you may be misunderstanding what deterministic means - or I am.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#16

Post by yorwba »

chairhairair wrote: post #12
joshuanapoli wrote: post #10

Earlier quoted context omitted.

I think that the hope/dream here is to make end-to-end tests less flakey. It would be great to have navigation and assertions commands that are robust against simple changes in the app that aren't relevant to the test case.

It's just a dream then. It's completely at-odds with the strengths of LLMs (fuzzy associations, rough summaries, naive co-thinking).

Fuzzy associations seem relevant? Interact with the UI based on what it looks like, not the specific implementation details.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#17

Post by MattDaEskimo »

digging wrote: post #14
aksophist wrote: post #7

Earlier quoted context omitted.

It’s only deterministic for each version of the app. Versions change: UI elements move, change their title slightly. Irrelevant promo popups appear, etc. For a deterministic solution, someone has to go and update the tests to handle all of that. Good ‘accessibility hygiene’ can help, but many apps lack that. And then there are truly dynamic apps like games or simulators. There may be no accessibility info to determin…

> Versions change: UI elements move, change their title slightly Not randomly, I'd hope. I think you may be misunderstanding what deterministic means - or I am.

It's crazy to have people so out of their league try to argue against well established meanings.

A testing framework requires determinism. If something changes the team should know and adjust.

AI could play a bit in easing this adjustment and tests but it's not a driver in these tests.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#18

Post by fullstackchris »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

To be fair, this is NOT the case with native mobile apps. There are some projects like detox that are trying to make e2e tests easier, but the tests themselves can be painful, run fairly slow on emulators, etc.

Maybe someday the tooling for mobile will be as good as headless chrome is for web :)

Agreed though that the followup debugging of a failed test could be hard to automate in some cases.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#20

Post by cschiller »

101008 wrote: post #4

Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.

I agree that it can seem counterintuitive at first to apply LLM solutions to testing. However, in end-to-end testing, we’ve found that introducing a level of flexibility can actually be beneficial.

Take, for example, scenarios involving social logins or payments where external webviews are opened. These often trigger cookie consent forms or other unexpected elements, which the app developer has limited control over. The complexity increases when these elements have unstable identifiers or frequently changing attributes. In such cases, even though the core functionality (e.g., logging in) works as expected, traditional test automation often fails, requiring constant maintenance.

The key, as to other comments, is ensuring the solution is good at distinguishing between meaningful test issues and non issues.

Post reply on HN