Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#71

Post by epolanski »

batikha wrote: post #32

Very cool! I already can see a lot of "this is already solved by playwright/cypress/selenium/deterministic stuff" in the comments. Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a goo…

> yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA

Because software is a clownish mimicking of engineering that lacks any real solid and widespread engineering practices.

It's cultural.

Crowds boast their engineering degrees, but have little to show but leetcode and system design black belts, even though their day to day job rarely requires them to architect systems or reimplement a new Levehnstein distance but would benefit a lot from thoroughly investigating functional and non functional requirements and encoding and maintaining those through automation.

There's very little engineering in software, people really care about the borderline fun parts and discard the rest.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#72

Post by xyst »

I remember testing out a similar product (mabl?). Ended up just using it to check for dead links. Using it for other use cases, I remember getting too many false positives for other use cases.

This was many years ago though (2018-2019?) before the genAI craze. Wonder if it has improved or not; or if this product is any better than its competitors.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#73

Post by ergeysay »

msoad wrote: post #67
codedokode wrote: post #58

Earlier quoted context omitted.

Sorry if it is a stupid idea, but cannot you log all messages to a separate file for each test (or attach test id to the messages)? Then if the test fails, you can see where the error occured.

Where I work there are 1,500 microservices. How do I get a log of all of those services -- only related to my test's requests in a file? I know there are solutions for this, but in the real world I have not seen it properly implemented.

As you said, OpenTelemetry and friends can help. I had great success with these.

I am curious, what were implementation issues you have encountered?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#74

Post by AdieuToLogic »

tomatohs wrote: post #44
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

You're totally right here, but "debugging failed tests" is a mature problem that assumes you have working tests and people to write them. Most companies don't have the resources to dedicate full engineer time to QA, and if they do nobody maintains the test. Debugging failed test is a "first world problem"

> ... "debugging failed tests" is a mature problem that assumes you have working tests and people to write them.

I am reminded of an old s/w engineering law:

  Developers can test their solution or Customers will.
  Either way, the system will be tested.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#75

Post by LeFever »

tauntz wrote: post #61
doublerebel wrote: post #56

How does this compare with Test.ai (now aka Testers.ai) who have offered basically this same service for the last 5 years?

Totally offtopic but I looked at testers.ai and noticed the following from the terms of service: > Individuals with the last name "Bach" or "Bolton" are prohibited from using, referencing, or commenting on this website or any of its content. ..and now I'm curious to know the backstory for this :)

John Bolton and James Bach are the founders of RST [1] and generally big names in the “formal” software testing space. Presumably the testers.ai folks aren’t fans. :p

[1] https://rapid-software-testing.com/authors/

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#76

Post by edelans »

fullstackchris wrote: post #18
msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

To be fair, this is NOT the case with native mobile apps. There are some projects like detox that are trying to make e2e tests easier, but the tests themselves can be painful, run fairly slow on emulators, etc. Maybe someday the tooling for mobile will be as good as headless chrome is for web :) Agreed though that the followup debugging of a failed test could be hard to automate in some cases.

I think we can claim that at Waldo.

Check for yourself: I've just recorded this [1] scripted test on the wikipedia mobile app, and it yields this [2] Replay. In less than a minute we spin up a fresh virtual device, install your app on it, execute the 8 steps of the script.

As a result, you get the Replay of the session : video synchronized with interaction timeline, device & network logs, so you can debug in full context.

[1]: https://github.com/waldoapp/waldo-programmatic-samples/blob/... [2]: https://share.waldo.com/7a45b5bd364edbf17c578070ce8bde220240...

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#77

Post by edelans »

cschiller wrote: post #57
mmaunder wrote: post #35

Congrats! How has Anthropic's latest release supporting computer use affected your planning/thinking around this? PS:If you had this for desktop we'd immediately become a customer.

Thank you! Sonnet 3.5 is indeed a powerful model, and we're actually using it. However, even with the latest version, there are still some limitations affecting our specific use case. For instance, the model struggles to accurately recognize semi-overlaid areas, such as popups that block interactions, and it has trouble consistently detecting when UI elements are in a disabled state. To address these issues, we enhan…

Hi cschiller, I think we can help you with those issues at Waldo. I guess you are using Appium under the hood to get the UI hierarchy. At Waldo we developed a competing (proprietary) engine that solves a lot of Appium problems.

We provide the most accurate view hierarchy for mobile apps (including React Native and Flutter apps), and we do it under 500ms for each view.

I would love to get in touch: at e.de-lansalut [at] tricentis.com

Here is an example of what we are able to do: https://share.waldo.com/7a45b5bd364edbf17c578070ce8bde220240...

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#78

Post by antonvs »

msoad wrote: post #67
codedokode wrote: post #58

Earlier quoted context omitted.

Sorry if it is a stupid idea, but cannot you log all messages to a separate file for each test (or attach test id to the messages)? Then if the test fails, you can see where the error occured.

Where I work there are 1,500 microservices. How do I get a log of all of those services -- only related to my test's requests in a file? I know there are solutions for this, but in the real world I have not seen it properly implemented.

This works easily enough in the major cloud environments, since logging tends to be automatic and centralized. The only thing you need to do is make sure that a common request id or similar propagates to all the services, which is not that difficult.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#80

Post by aksophist »

chrtng wrote: post #64
aksophist wrote: post #21

how do you evaluate your tool, and have you published your evaluation along with the metrics?

Thank you for your question! While we haven't published a formal evaluation yet, it's something we are working toward. Currently, we rely mostly on human reviews to monitor and assess LLM outputs. We also maintain a golden test suite that is run against every release to ensure consistency and quality over time, using regex-based evaluations. Our key metrics include the time and cost per agentic loop, as well as the f…

What is a false positive rate? Is it when the agent falsely passes or falsely “finds a bug”? And regardless of which: why don’t you include the other as a key metric?

I’m not aware of any evals or shared metrics. But measuring a testing agents performance seems pretty important.

What is your tool’s FPR on your golden suite?

Post reply on HN