Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#31

Post by Lienetic »

hn_throwaway_99 wrote: post #27
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

While I agree with your primary pain point, I would argue that that really isn't specific to tests at all. It sounds like what you're really saying is that when something goes wrong, it's really difficult to determine which component in a complex system is responsible. I mean, from what you've described (and from what I've experienced as well), you would have the same if not harder problem if a user experienced a bug…

Agreed. Is there a good tool you'd recommend for this?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#32

Post by batikha »

Very cool! I already can see a lot of "this is already solved by playwright/cypress/selenium/deterministic stuff" in the comments.

Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a good thing), also on happy path / core features (arguably a waste of time to test things over and over again).

More than once I worked for a company that was against having a manual QA team, out of principle and more or less valid reasons (we use a typed language so less bug, engineers are empowered, etc etc), but ended up hiring external consultants to handle QA after a big quality incident.

The amount of mismatch between theory and practice in this field is impressive.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#33

Post by conorjh »

usernameis42 wrote: post #22
chairhairair wrote: post #8

Earlier quoted context omitted.

This fundamental issue seems to be totally lost on the LLM-heads. I do not want additional uncertainty deep in the development cycle. I can tolerate the uncertainty while I'm writing. That's where there is a good fit for these fuzzy LLMs. Anything past the cutting room floor and you are injecting uncertainty where it isn't tolerable. I definitely do not want additional uncertainty in production. That's where the "lar…

Same goes with the real people, we all can do mistakes and AI Agents would get better over time, and will be ahead of many specialist pretty soon, but probably not perfect before AGI, just as we are.

your software has real people in it?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#34

Post by hn_throwaway_99 »

Lienetic wrote: post #31
hn_throwaway_99 wrote: post #27

Earlier quoted context omitted.

While I agree with your primary pain point, I would argue that that really isn't specific to tests at all. It sounds like what you're really saying is that when something goes wrong, it's really difficult to determine which component in a complex system is responsible. I mean, from what you've described (and from what I've experienced as well), you would have the same if not harder problem if a user experienced a bug…

Agreed. Is there a good tool you'd recommend for this?

It's been quite some time but New Relic is a popular observability tool whose primary goal (at least the original primary goal I'd say) is being able to tie together lots of distributed systems to make it easier to do request tracing and root cause analysis. I was a big fan of New Relic when I last used it, but if memory serves me correctly it was quite expensive.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#36

Post by codetrotter »

cschiller wrote: post #13
codepathfinder wrote: post #2

I've been a mobile developer for the past 10 years and my overall belief is that mobile app development has slower growth and companies with the mobile team are investing less on mobile Dev or testing+tooling+education. Do you think the market is still hot once it was to use your product?

I would say that mobile apps are still the primary format for launching new consumer services, incl. new apps like ChatGPT and many others. However we’ve observed that teams are expected to do more with less—delivering high-quality products while ensuring compliance, often with the same or even smaller team sizes. This is why we focus on minimizing the engineering burden, particularly when it comes to repetitive task…

> mobile apps are still the primary format for launching new consumer services, incl. new apps like ChatGPT and many others

OpenAI launched ChatGPT to the public on the web first and it took like, several months I think from I used their public web version until they had an official app for it in App Store. In the meantime, some third party apps popped up in App Store for using ChatGPT. I kept using the web version until the official app showed up. And probably having the mobile app in App Store has helped them grow to the number of users they have now. But IMO, ChatGPT as a product was not itself “launched” on App Store and they seemed to do very well in terms of adoption even when initially they only had the web version. The main point, that mobile apps are still desired, I agree with though.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#37

Post by minhaz23 »

digging wrote: post #14
aksophist wrote: post #7

Earlier quoted context omitted.

It’s only deterministic for each version of the app. Versions change: UI elements move, change their title slightly. Irrelevant promo popups appear, etc. For a deterministic solution, someone has to go and update the tests to handle all of that. Good ‘accessibility hygiene’ can help, but many apps lack that. And then there are truly dynamic apps like games or simulators. There may be no accessibility info to determin…

> Versions change: UI elements move, change their title slightly Not randomly, I'd hope. I think you may be misunderstanding what deterministic means - or I am.

Ever worked with extjs? :/

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#38

Post by devjab »

101008 wrote: post #4

Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.

I think it’s less of an issue for e2e testing because e2e testing sucks. If teams did it well in general you would be completely correct, but in many places a LLM will be better even if it hallucinates. As such I think there will be a decent market for products like this, even if they aren’t may not even really be testing what you think they are testing. Simply because that may well be way better than the e2e testing many places already do.

In many cases you’re correct though. We have a few libraries where we won’t use Typescript because even though it might transpire 99% correctly, the fact that we have to check, is too much work for it to be worth our time in those cases. I think LLMs are similar, once in a while you’re not going to want them because checking their work takes too much resources, but for a lot of stuff you can use them. Especially if your e2e testing is really just pseudo jobbing because some middle manager wanted it, which it unfortunately is far too often. If you work in such a place you’re going to recommend the path of least resistance and if that’s LLM powered then it’s LLM powered.

On the less bleak and pessimistic side, if the LLM e2e output is good enough to be less resource consuming, even if you have to go over it, then it’s still a good business case.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#39

Post by cschiller »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

Thanks for your thoughtful response! Agree that digging into the root cause of a failure, especially in complex microservice setups, can be incredibly time-consuming.

Regarding writing robust e2e tests, I think it really depends on the team's experience and the organization’s setup. We’ve found that in some organizations—particularly those with large, fast-moving engineering teams—test creation and maintenance can still be a bottleneck due to the flakiness of their e2e tests.

For example, we’ve seen an e-commerce team with 150+ mobile engineers struggle to keep their functional tests up-to-date while the company was running copy and marketing experiments. Another team in the food delivery space faced issues where unrelated changes in webviews caused their e2e tests to fail, making it impossible to run tests in a production-like system.

Our goal is to help free up that time so that teams can focus on solving bigger challenges, like the debugging problems you’ve mentioned.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#40

Post by rafaelmn »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

I think either you're overselling the maturity of the ecosystem or I've been unfortunate enough to get stuck with the worst option out there - Cypress. I run into tooling limitations and issues regularly, only to eventually find an open GitHub issue with no solution or some such.
Post reply on HN