how do you evaluate your tool, and have you published your evaluation along with the metrics?
Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
21–30 of 84 posts
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#22Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.
This fundamental issue seems to be totally lost on the LLM-heads. I do not want additional uncertainty deep in the development cycle. I can tolerate the uncertainty while I'm writing. That's where there is a good fit for these fuzzy LLMs. Anything past the cutting room floor and you are injecting uncertainty where it isn't tolerable. I definitely do not want additional uncertainty in production. That's where the "lar…
Same goes with the real people, we all can do mistakes and AI Agents would get better over time, and will be ahead of many specialist pretty soon, but probably not perfect before AGI, just as we are.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#23Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.
I work in the field and built a tool that has way less flakiness than deterministic solutions.
The issue is testing environments are always imperfect because (a) they are stateful and (b) there's always some randomness in actual production software. Some teams have very clean testing environment but most don't.
So being non-deterministic is actually an advantage, in practice.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#24Earlier quoted context omitted.
It's just a dream then. It's completely at-odds with the strengths of LLMs (fuzzy associations, rough summaries, naive co-thinking).
Fuzzy associations seem relevant? Interact with the UI based on what it looks like, not the specific implementation details.
No. Both of the requirements "to interact" and "based on what it looks like" require unshakable foundations in reality - which current models clearly do not have.
They will inevitably hallucinate interactions and observations and therefore decrease reliability. Worse, they will inject a pervasive sense of doubt into the reliability of any tests they interact with.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#25I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…
I doubt that screenshot methods are the bottleneck considering that's the method Microsoft and Anthropic are using.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#26Still interesting how a lot of companies offer a LLM (non-deterministic) solution for deterministic problems.
It’s only deterministic for each version of the app. Versions change: UI elements move, change their title slightly. Irrelevant promo popups appear, etc. For a deterministic solution, someone has to go and update the tests to handle all of that. Good ‘accessibility hygiene’ can help, but many apps lack that. And then there are truly dynamic apps like games or simulators. There may be no accessibility info to determin…
There is great approach based on test-id strategy, basically it's a requirement for the frontend teams to cover all interactive elements with test-id's.
It allows to make tests less flaky and writing them is increasing dramatically, also works with mobile as well, usually elements for the main flows doesn't change that often, you'll still need to update them.
I did stable mobile UI tests with this approach as well, worked well
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#27I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…
While I agree with your primary pain point, I would argue that that really isn't specific to tests at all. It sounds like what you're really saying is that when something goes wrong, it's really difficult to determine which component in a complex system is responsible. I mean, from what you've described (and from what I've experienced as well), you would have the same if not harder problem if a user experienced a bug on the front end and then you had to find the root cause.
That is, I don't think a framework focused on front end testing should really be where the solution for your problem is implemented. You say "This is a very, very difficult thing to automate and requires AGI-level intelligence to really build a system that can go read the logs of some random service deep in our service mesh to understand why an e2e test fails." - I would argue what you really need is better log aggregation and system tracing. And I'm not saying this to be snarky (at scale with a bunch of different teams managing different components I've seen that it can be difficult to get everyone on the same aggregation/tracing framework and practices), but that's where I'd focus, as you'll get the dividends not only in testing but in runtime observability as well.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#28How does this compare to Robin by mobile.dev; the same guys that built Maestro? [0] That has around 95% of what GPT Driver does and has the potential to do Web E2E testing. [0] https://maestro.mobile.dev
One of our customers recently compared GPTD with Maestro’s Robin (formerly App Quality CoPilot). Their mobile platform engineering manager highlighted three key reasons for choosing us: lack of frustration, ease of implementation, and reliability.
To be more concrete their words were: - “What you define, you can tweak, touch the detail, and customize, saving you time.” - “You don’t entirely rely on AI. You stay involved, avoiding misinterpretations by AI.” - “Flexibility to refine, by using templates and triggering partial tests, features that come from real-world experience. This speeds up the process significantly.”
Our understanding is that because we launched the first version of GPT Driver in April 2023, we’ve built it in an “AI-native” way, while other tools are simply adding AI-based features on top. We worked closely with leading mobile teams, including Duolingo, to ensure we stay as aligned as possible with real-world challenges.
While our focus is on mobile, GPT Driver also works effectively on web platforms.
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#29Curious question, what ever happened with the OpenAI drama with trademarking “GPT”. I’m guessing they were not successful?
Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
#30Curious question, what ever happened with the OpenAI drama with trademarking “GPT”. I’m guessing they were not successful?
From what we understand the term GPT was deemed too general for OpenAI to claim as its own.
https://www.theverge.com/2024/2/16/24075304/trademark-pto-op...