Awesome idea! Excited to see where this goes
Show HN: Checksum – generate and maintain end-to-end tests using AI
21–30 of 34 posts
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#22This is dope, a couple of feedback points from a technical person that is a potential customer (I could be wrong on these!):
-- I think the name doesn't sell me or even most people because "checksum" is more of a security/crypto term. When I saw the HN post say Checksum I didn't think it was going to be about end-to-end tests. I thought it was going to be some crypto thing. Maybe a name like "Tested" or "Covered" is going to click better with the potential customer.
-- I don't feel like the demo video is making me feel like I know what this product is doing. I could also be misunderstanding the product. It might help more if the demo showed the following (in ideally less than 5-10s or most users might tune out):
1. A quick setup step for checksum 2. A set of generated tests 3. Passing tests
Seeing those steps would give me the emotion as an end-user "wow this must something I can quickly setup and will make me feel like I have test coverage out of the box"
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#23This is dope, a couple of feedback points from a technical person that is a potential customer (I could be wrong on these!): -- I think the name doesn't sell me or even most people because "checksum" is more of a security/crypto term. When I saw the HN post say Checksum I didn't think it was going to be about end-to-end tests. I thought it was going to be some crypto thing. Maybe a name like "Tested" or "Covered" is…
Thanks for your feedback. It definitely makes sense and we'll incorporate it!
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#24Awesome idea! Excited to see where this goes
Thank you! Appreciate the kind words.
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#25Earlier quoted context omitted.
That's a great question! We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible. We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.
If this is trained on user sessions, how would the model learn to generate tests for edge cases that wouldn’t necessarily show up in the training data?
We train the model based on user sessions to learn how to use an app. The model learns how to execute specific flows, but also how to interact with components in the more general sense. Since most developers use compostable components, patterns of usage are repeated across the same app.
Then, during test generation, we bias the model to explore edge cases (in a few ways), and the model is still able to complete those even with low sample.
In other words, we direct the model toward certain goals, and flows and also add chaos to the process which result in the model executing unexpected flows.
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#26Nothing on pricing on your site. Makes any other question difficult to formulate.
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#27I'm always suspicious of tests when test coverage is the main metric. I've seen developers write tests that don't really check anything but run all the code paths. I've also seen tests that check every bit of output, which end up being brittle. How well do the tests hold up over time, and how well are the tests validating the contract of the code instead of just historical behavior and quirks?
That's a great question! We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible. We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.
How do you know what data is sensitive, and how do you anonymize?
Thinking of apps that might fall under HIPAA etc
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#28Nothing on pricing on your site. Makes any other question difficult to formulate.
Yep totally understand. We are an early stage startup and currently 100% focused on improving our models and our product.
We don’t have pricing, not because we try to be vague, but because we haven’t fully figured out our training costs, which can vary significantly per app. We are very much in the “Do things that don’t scale” phase where we hand-pick our customers, provide white-glove treatment and prioritize learnings over price
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#29Nothing on pricing on your site. Makes any other question difficult to formulate.
Yep totally understand. We are an early stage startup and currently 100% focused on improving our models and our product. We don’t have pricing, not because we try to be vague, but because we haven’t fully figured out our training costs, which can vary significantly per app. We are very much in the “Do things that don’t scale” phase where we hand-pick our customers, provide white-glove treatment and prioritize learni…
Reasonable.
Re: Show HN: Checksum – generate and maintain end-to-end tests using AI
#30Earlier quoted context omitted.
That's a great question! We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible. We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.
How do you know what data is sensitive, and how do you anonymize? Thinking of apps that might fall under HIPAA etc
We have a few privacy controls in place:
1. We hash all inner text and then backfill static strings on the server side. So every text that is specific to the user remains hashed
2. We detect special cases like passwords, SSNs, credit cards, and completely block it (even not hashed)
3. We provide full privacy controls to our customers to easily mask any sensitive elements
4. We discard the user IP and don't require any PII to be sent. So we can connect a session together, but don't really know who the user is