Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: CamelQA (YC W24) – AI that tests mobile apps

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 55 posts

Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#1

Post by vercantez »

Hey HN! We're camelQA (https://camelqa.com/). We’re building an AI agent that can automate mobile devices using computer vision. Our first use case is for mobile app QA. We convert natural language test cases into tests that run on real iOS and Android devices in our device farm.

Flaky UI tests suck. We want to create a solution where engineers don’t waste time maintaining fragile scripts.

camelQA uses a combination of accessibility element data along with an in-house custom vision-only RCNN object detection model paired with Google Siglip for UI element classification (see a sample output here - https://camelqa.com/blog/sole-ui-element-detector.png). This way we’re able to detect elements even if they do not have accessibility elements associated with them.

Under the hood the agent is using Appium to interface with the device. We use GPT-4V to reason at a high level and GPT-3.5 to execute the high-level actions. Check out a gif of our playground here (https://camelqa.com/blog/sole-signup.gif)

Since we’re vision based, we don’t need access to your source code and we work across all app types - SwiftUI and UIKit, React Native, Flutter.

We built a demo for HN where you can use our model to control Wikipedia on a simulated iPhone. Check that out here (https://demo.camelqa.com/). Try giving it a task like “Bookmark the wiki page for Ilya Sutskever“ or “Find San Francisco in the Places tab”. We only have 5 simulators running so there may be a wait. You get 5 minutes once you enter your first command.

If you want to see what our front end looks like, we made an account with some test runs. Use this login (Username: hackerNews Password: 1337hackerNews!) to view our sandboxed HN account (https://dash.camelqa.com/login).

Last year we left our corporate jobs to build in the AI space. It felt like we were endlessly testing our apps, even for minor updates, and we still shipped a bug that caused one of our apps to crash on subscribe (the app in question - https://apps.apple.com/us/app/tldr-ai-summarizer/id644930471...). That was the catalyst for camelQA.

We’re excited to hear what you all think!

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#2

Post by bluelightning2k »

Very cool! I don't have this pain point currently but I can absolutely see the utility. I like the in built demo tool (although it sadly means you have no need for DemoTime lol).

The demo.camelqa needs some styling. I would invest a few minutes here. Maybe a loading spinner too if you're expecting 15second latency.

Technically is this doing clever things with markup, or literally just feeding the image into a multimodal LLM and getting function calls in response?

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#3

Post by vercantez »

bluelightning2k wrote: post #2

Very cool! I don't have this pain point currently but I can absolutely see the utility. I like the in built demo tool (although it sadly means you have no need for DemoTime lol). The demo.camelqa needs some styling. I would invest a few minutes here. Maybe a loading spinner too if you're expecting 15second latency. Technically is this doing clever things with markup, or literally just feeding the image into a multimo…

Thanks for the feedback! We'll add some styling to the demo page. We're processing the image with an object detection model and classification model and also using some accessibility element data to get a better understanding of what is interactive on the screen.

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#4

Post by hubraumhugo »

As someone who worked in a mobile dev team, I can only applaud your effort! :)

I have a few questions:

As with all new AI-based RPA & Testing frameworks (there are quite many in YC), I'm curious about the costs and performance. Let's say I want to run a few smoke tests (5-10 end-to-end scenarios) on my app across multiple iOS and Android devices with different screen sizes and OS versions before going into production.

What would it cost, and how long would it take to complete the tests?

Do you already have customers running such real-world use cases with it?

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#6

Post by vercantez »

hubraumhugo wrote: post #4

As someone who worked in a mobile dev team, I can only applaud your effort! :) I have a few questions: As with all new AI-based RPA & Testing frameworks (there are quite many in YC), I'm curious about the costs and performance. Let's say I want to run a few smoke tests (5-10 end-to-end scenarios) on my app across multiple iOS and Android devices with different screen sizes and OS versions before going into production…

Good questions. Execution cost is indeed higher than traditional testing automation scripts but much lower than the human cost of writing and maintaining the scripts. We're starting at $500/month and our plans go up from there depending on how many devices you want to test across. We do have customers running across multiple devices and OS versions today.

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#7

Post by ck_one »

vercantez wrote: post #3
bluelightning2k wrote: post #2

Very cool! I don't have this pain point currently but I can absolutely see the utility. I like the in built demo tool (although it sadly means you have no need for DemoTime lol). The demo.camelqa needs some styling. I would invest a few minutes here. Maybe a loading spinner too if you're expecting 15second latency. Technically is this doing clever things with markup, or literally just feeding the image into a multimo…

Thanks for the feedback! We'll add some styling to the demo page. We're processing the image with an object detection model and classification model and also using some accessibility element data to get a better understanding of what is interactive on the screen.

Why don't you also use GPT4-V for that part?

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#8

Post by maxconf »

Having worked on mobile infra for many years now for a couple very large iOS teams, excited to learn more and kudos for putting yourselves out there. 1. Integration tests are notoriously slow, the demo seemed to take some time to do basic actions; is it even possible to run these at scale? 2. >Flaky UI tests suck; they can be flaky but it's often due to bad code and architecture. Any data to backup your tool makes the tests less flaky? I could see a scenario where there are 2 buttons with the same text, but under the hood we'd use different identifiers in-code to determine which button should be tapped in UI.

Overall I'm a bit skeptical because most UI tests are pretty easy to write today with very natural DSLs that are close to natural language, but definitely want to follow and hear more production use cases.

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#9

Post by vercantez »

ck_one wrote: post #7
vercantez wrote: post #3

Earlier quoted context omitted.

Thanks for the feedback! We'll add some styling to the demo page. We're processing the image with an object detection model and classification model and also using some accessibility element data to get a better understanding of what is interactive on the screen.

Why don't you also use GPT4-V for that part?

GPT-4V is great for reasoning about what is on the screen. However, it struggles with precision. For example, it is not able to specify the coordinates to tap when it decides to tap an icon. That's where the object detection and accessibility elements help. We can precisely locate interactive elements.

Re: Launch HN: CamelQA (YC W24) – AI that tests mobile apps

#10

Post by bluelightning2k »

vercantez wrote: post #9
ck_one wrote: post #7

Earlier quoted context omitted.

Why don't you also use GPT4-V for that part?

GPT-4V is great for reasoning about what is on the screen. However, it struggles with precision. For example, it is not able to specify the coordinates to tap when it decides to tap an icon. That's where the object detection and accessibility elements help. We can precisely locate interactive elements.

Have you tried putting a pixel grid over the image with labelled guidelines every 100px?

Was one thing I never got around to testing with DemoTime but was always curious about.

Anyway sorry this is a nice product. Congratulations on the launch.

Always good to see substantial tech

Post reply on HN