Your demo is very concise and well crafted. Is your host naturally smooth or it was many takes? Good job
Launch HN: CamelQA (YC W24) – AI that tests mobile apps
11–20 of 55 posts
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#12Having worked on mobile infra for many years now for a couple very large iOS teams, excited to learn more and kudos for putting yourselves out there. 1. Integration tests are notoriously slow, the demo seemed to take some time to do basic actions; is it even possible to run these at scale? 2. >Flaky UI tests suck; they can be flaky but it's often due to bad code and architecture. Any data to backup your tool makes th…
Great questions.
1. Yes, running tests in parallel helps. We also cache actions so subsequent runs are much faster (this is disabled in the demo).
2. I agree that testing can be much more reliable and pleasant in some codebases than others. I have not been blessed with these types of codebases in my career. Flakiness is from personal experience automating UI tests specifically and having them break when a new nondeterministic popup modal is added or another engineer breaks an identifier/locator strategy.
That being said, if you like writing UI tests and your codebase supports easily maintaining them, there are some really cool DSLs like Maestro!
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#13Earlier quoted context omitted.
GPT-4V is great for reasoning about what is on the screen. However, it struggles with precision. For example, it is not able to specify the coordinates to tap when it decides to tap an icon. That's where the object detection and accessibility elements help. We can precisely locate interactive elements.
Have you tried putting a pixel grid over the image with labelled guidelines every 100px? Was one thing I never got around to testing with DemoTime but was always curious about. Anyway sorry this is a nice product. Congratulations on the launch. Always good to see substantial tech
Thanks! Yes, we experimented with that! I think because of the way that GPT sees images in patches it has a hard time with absolute positioning but that's just a guess.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#14Your demo is very concise and well crafted. Is your host naturally smooth or it was many takes? Good job
Our host is naturally smooth. We've tried a few different platforms to host this demo and landed on mac stadium. Glad you enjoyed your experience.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#15Having worked on mobile infra for many years now for a couple very large iOS teams, excited to learn more and kudos for putting yourselves out there. 1. Integration tests are notoriously slow, the demo seemed to take some time to do basic actions; is it even possible to run these at scale? 2. >Flaky UI tests suck; they can be flaky but it's often due to bad code and architecture. Any data to backup your tool makes th…
Great questions. 1. Yes, running tests in parallel helps. We also cache actions so subsequent runs are much faster (this is disabled in the demo). 2. I agree that testing can be much more reliable and pleasant in some codebases than others. I have not been blessed with these types of codebases in my career. Flakiness is from personal experience automating UI tests specifically and having them break when a new nondete…
> We also cache actions so subsequent runs are much faster
Interesting, what do you cache? How do you know if 1 change needs to be rerun versus another?
>Flakiness is from personal experience automating UI tests specifically and having them break when a new nondeterministic popup modal is added or another engineer breaks an identifier/locator strategy
A modal popping up isn't a flake though, it's often when a button is on screen but the test runner can't seem to find it due to run-loop issues or emulator/simulator issues. If a modal pops up on the screen in a test, how does CamelQA resolve this and how would it know if it's an actual regression or not? If a modal pops up on a screen at the wrong time that _could_ be a real regression, versus a developer forgetting to configure some local state.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#16Earlier quoted context omitted.
Great questions. 1. Yes, running tests in parallel helps. We also cache actions so subsequent runs are much faster (this is disabled in the demo). 2. I agree that testing can be much more reliable and pleasant in some codebases than others. I have not been blessed with these types of codebases in my career. Flakiness is from personal experience automating UI tests specifically and having them break when a new nondete…
> We also cache actions so subsequent runs are much faster Interesting, what do you cache? How do you know if 1 change needs to be rerun versus another? >Flakiness is from personal experience automating UI tests specifically and having them break when a new nondeterministic popup modal is added or another engineer breaks an identifier/locator strategy A modal popping up isn't a flake though, it's often when a button…
1. The AI agent writes an automation script (similar to Appium) that we can replay after the first successful run. If there are issues the AI agent gets pulled back into the loop.
2. You can define acceptance criteria in natural language with camel.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#17Having worked on mobile infra for many years now for a couple very large iOS teams, excited to learn more and kudos for putting yourselves out there. 1. Integration tests are notoriously slow, the demo seemed to take some time to do basic actions; is it even possible to run these at scale? 2. >Flaky UI tests suck; they can be flaky but it's often due to bad code and architecture. Any data to backup your tool makes th…
> most UI tests are pretty easy to write today with very natural DSLs that are close to natural language
Wouldn't it be a better/cheaper/faster solution to use LLMs to write UI/integration tests?
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#18Any support for non-mobile native apps? e.g. macOS?
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#19Having worked on mobile infra for many years now for a couple very large iOS teams, excited to learn more and kudos for putting yourselves out there. 1. Integration tests are notoriously slow, the demo seemed to take some time to do basic actions; is it even possible to run these at scale? 2. >Flaky UI tests suck; they can be flaky but it's often due to bad code and architecture. Any data to backup your tool makes th…
> most UI tests are pretty easy to write today with very natural DSLs that are close to natural language Wouldn't it be a better/cheaper/faster solution to use LLMs to write UI/integration tests?
The issue with this approach is that for all but the most simple apps it is not possible to deduce the runtime element information needed to write traditional UI tests given just the source code. This can only be done reliably at runtime which is what we do. We run your app and iteratively build UI tests that can be reused later.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#20Any support for non-mobile native apps? e.g. macOS?
Not yet but it's on the road map! We have a monthly newsletter if you'd like to stay up to date