11–20 of 34 posts
Does it only support web? What about react native mobile apps?
There's nothing "specific" in the underlying model that prevents it from testing mobile. It's just a matter of focus at the current time.
> Our impact on pefromence is non-existant as we use battle-tested open source tools used by Fortune 500 companies
Noticed a couple small typos in the marketing copy > Our impact on pefromence is non-existant as we use battle-tested open source tools used by Fortune 500 companies
Is it for the identification of user sessions that are good candidates to make into tests? Is it the generation of test specification in some DSL / Cucumber / Selenium / etc.?
What does that mean, exactly? Just because it's open source and used by F500s doesn't mean it can't have performance issues.
> Our impact on performance is non-existent as we use battle-tested open source tools used by Fortune 500 companies What does that mean, exactly? Just because it's open source and used by F500s doesn't mean it can't have performance issues.
1. We (and others) have tested our tools' impact on memory, CPU, network performance and found only negligible impact, even on slower/older devices
2. Also, they are used by F500 companies and have wide adoption, which indicates that other well established devs have run the same tests and decided to move forward.
We'll work on the language there to clarify.
How is the product different from the other test generation tools? How do you check if the are testing the intended behavior. My experience with automated testing solutions has been lukewarm so far.
I agree! My experience with test generation tools was also lukewarm which is why we founded Checksum. > How is the product different from the other test generation tools We train our models based on real user sessions. So our tests are: 1. Completely auto-generated 2. Achieve high coverage of real user flows, including detecting edge cases 3. Automatically maintained and execute with our models so they are less flake…
Congrats! Definitely think QA farms can be automated using AI! Can you explain more what part Checksum is using AI? Is it for the identification of user sessions that are good candidates to make into tests? Is it the generation of test specification in some DSL / Cucumber / Selenium / etc.?
1. We use AI to analyze the user patterns and find common paths and edge cases, basically building a representation of your UX in a DB
2. We then use the DB to train another ML model that learns how to use your app the same way a user does. Given a certain page and user context, the ML can complete UX flows.
3. Finally, we learn to generate assertions, run the tests and convert the model actions in step 2 into proper Playwright or Cypress tests
I'm always suspicious of tests when test coverage is the main metric. I've seen developers write tests that don't really check anything but run all the code paths. I've also seen tests that check every bit of output, which end up being brittle. How well do the tests hold up over time, and how well are the tests validating the contract of the code instead of just historical behavior and quirks?
That's a great question! We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible. We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.
Congratulations on all the progress you've made! We are all learning as we're building and talking to users. I know for my team, E2E/Integration testing is our main priority (over unit tests), and maintaining E2E tests is definitely a struggle. I imagine this problem is even more of an issue for larger codebases so I see why you're going after medium-size startups where the product isn't completely rebuilt every few…
Feel free to sign up for a demo if that's a priority for your team. Even if it's just to chat and connect.