Seems similar to App Quality Copilot - https://www.mobile.dev/app-quality-copilot
They're building something based on appium and specifically through Maestro which is an excellent use of appium. But it isn't AI - we're excited to try it out!
Launch HN: CamelQA (YC W24) – AI that tests mobile apps
41–50 of 55 posts
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#42would be nice to just invite the AI tester to TestFlight instead of uploading the build :)
noted:) thanks for the feedback I think that would be an easier way to kick off camelQA as well. We'll have to learn how to do that in our remote physical device farm.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#43This is very cool. Is there a version for browsers?
Thanks. It's on our roadmap, but we're starting with mobile first.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#44Yeah, the two big issues with UI tests: flaky and slow. Curious how using GPT and vision combats flakiness? I'd feel the entropy of GPT and anything less than 100% accuracy in the computer vision pieces would lead to more flakiness. I also wonder about the speed and costs of running the tests. When E2E tests are traditionally slow and expensive already. The computer vision and GPT elements seem costlier and less fast…
We use GPT 4V to reason about the screen and decide what to do next. It does make mistakes. Here's a video of it thinking a page in the shop app is an ad (https://www.youtube.com/watch?v=MKyO-U7j4Hs).
The upside is that we do prompt hacking on our end to break out of loops and heal after it's made a mistake. Having said that, we're working on improving this!
On costs, it's cheaper than you think. The entire playground demo cost us less than $10. More expensive than running a script but we believe the cost of intelligence will go down in time.
On speed, yes it is slow. We minimize this by parallelizing tests across devices on our device farm. We can normally turn results around in 2.5-4 hours depending on the number of tests.
Thanks for the questions!
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#45I love this idea. Is there something similar for web-apps?
I wonder if you can easily add AI-based fuzzing or AI-based sample workflows to a testing pipeline.
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#46I love this idea. Is there something similar for web-apps? I wonder if you can easily add AI-based fuzzing or AI-based sample workflows to a testing pipeline.
Thanks! Web app support is coming very soon. That would be a really cool application of the API
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#47This is a great idea. I assume that inference costs will be higher for the time being, but it does aim to solve a real problem. Kudos..!
Thanks! Yes inference costs are non-negligible right now but we think this will come down over time
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#48I love this idea. Is there something similar for web-apps? I wonder if you can easily add AI-based fuzzing or AI-based sample workflows to a testing pipeline.
Thanks! Web app support is coming very soon. That would be a really cool application of the API
I'd love to give an AI a list of workflows to try out every time I push an update to my site.
In addition to pass/fail, I can see it even leaving some comments about ease-of-use. There's a lot of value here!
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#49Why is the branding/mascot a camel?
I'm reminded of Waldo, a mobile testing automation product that was acquired in 2023.
Their mascot is another camelid (not sure if alpaca or llama). https://www.waldo.com/
Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps
#50Why is the branding/mascot a camel? I'm reminded of Waldo, a mobile testing automation product that was acquired in 2023. Their mascot is another camelid (not sure if alpaca or llama). https://www.waldo.com/
We originally spelled it qaml which stood for quality assurance, machine learning. That wasn't obvious to anyone and everyone would pronounce it as "qwamel" which we hated so we decided on camelQA instead. Thanks for the question. I was waiting for someone to ask it!