Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#51

Post by tomatohs »

codepathfinder wrote: post #3

Is it possible to record the user screen and just generate a test case. I believe that's most efficient way IMO

This comes up all the time. It seems like it would be possible, but imagine the case where you want to verify that a menu shows on hover. Was the hover on the menu intentional?

Another example, imagine an error box shows up. Was that correct or incorrect?

So you need to build a "meta" layer, which includes UI, to start marking up the video and end up in the same state.

Our approach has been to let the AI explore the app and come up with ideas. Less interaction from the user.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#52

Post by pj_mukh »

This is super cool. As a question, are the instructions re-generated from the instruction tokens everytime. While maybe costly, this feels like it would be robust to small changes in the app (and component name changes etc.). Does that make sense?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#53

Post by tomatohs »

chairhairair wrote: post #24
yorwba wrote: post #16

Earlier quoted context omitted.

Fuzzy associations seem relevant? Interact with the UI based on what it looks like, not the specific implementation details.

No. Both of the requirements "to interact" and "based on what it looks like" require unshakable foundations in reality - which current models clearly do not have. They will inevitably hallucinate interactions and observations and therefore decrease reliability. Worse, they will inject a pervasive sense of doubt into the reliability of any tests they interact with.

> unshakable foundations in reality

Yes, you are correct that it entirely lays in the reputation of the AI.

This discussion leads to interesting question, which is "what is quality?"

Quality is determined by perception. If we can agree that an AI is acting like a user and it can use your website, we can assume that a user can use your website and therefor it is "quality".

For more, read "Zen and the Art of Motorcycle Maintenance"

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#54

Post by layer8 »

usernameis42 wrote: post #22
chairhairair wrote: post #8

Earlier quoted context omitted.

This fundamental issue seems to be totally lost on the LLM-heads. I do not want additional uncertainty deep in the development cycle. I can tolerate the uncertainty while I'm writing. That's where there is a good fit for these fuzzy LLMs. Anything past the cutting room floor and you are injecting uncertainty where it isn't tolerable. I definitely do not want additional uncertainty in production. That's where the "lar…

Same goes with the real people, we all can do mistakes and AI Agents would get better over time, and will be ahead of many specialist pretty soon, but probably not perfect before AGI, just as we are.

One of the advantages of automation has traditionally been that it cuts out the indeterminacy and variability inherent in real people.

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#55

Post by SkyBelow »

conorjh wrote: post #33
usernameis42 wrote: post #22

Earlier quoted context omitted.

Same goes with the real people, we all can do mistakes and AI Agents would get better over time, and will be ahead of many specialist pretty soon, but probably not perfect before AGI, just as we are.

your software has real people in it?

Ideally it does. Users, super users, admins, etc. Though one might point out exactly how much effort we put into locking down what they can do. I think one might be able to expand this to build up a persona for how LLMs should interface with software in production, but too many applications give them about the same level of access as a developer coding straight into production. Then again, how many company leaders would approve of that as well if they thought it would get things done faster and at lower cost?

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#57

Post by cschiller »

mmaunder wrote: post #35

Congrats! How has Anthropic's latest release supporting computer use affected your planning/thinking around this? PS:If you had this for desktop we'd immediately become a customer.

Thank you! Sonnet 3.5 is indeed a powerful model, and we're actually using it. However, even with the latest version, there are still some limitations affecting our specific use case. For instance, the model struggles to accurately recognize semi-overlaid areas, such as popups that block interactions, and it has trouble consistently detecting when UI elements are in a disabled state.

To address these issues, we enhance the models with our own custom logic and specialized models, which helps us achieve more reliable results.

Looking forward, we expect our QA Studio to become even more powerful as we integrate tools like test management, reporting, and infrastructure, especially as models improve. We're excited about the possibilities ahead!

Re: Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language

#58

Post by codedokode »

msoad wrote: post #9

I work in this space. We manage thousands of e2e tests. The pain has never been in writing the tests. Frameworks like Playwright are great at the UX. And having code editors like Cursor makes it even easier to write the tests. Now, if I could show Cursor the browser, it would be even better, but that doesn’t work today since most multimodal models are too slow to understand screenshots. It used to be that the fronten…

Sorry if it is a stupid idea, but cannot you log all messages to a separate file for each test (or attach test id to the messages)? Then if the test fails, you can see where the error occured.
Post reply on HN