Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: We built a tool that automatically generates API tests

stepci.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Show HN: We built a tool that automatically generates API tests

#41

Post by intelVISA »

itake wrote: post #25
Wissmueller wrote: post #11

Earlier quoted context omitted.

Hi, this is Sebastian from Step CI Thanks for sharing Venom! Certainly, there are lots of tools in the space, both SaaS and open-source. Mish and I can name you a few! However, there’s one thing these tools seem to be missing - a greater goal. Our vision is to give you a tool that will free you from writing and maintaining API tests ever again - in any shape or form. What you currently see from us is the first step!

> Venom is a CLI (Command Line Interface) that aim to create, manage and run your integration tests with efficiency. The linked repo seems to have a greater goal.

I'm torn, half of me wants to support them for the hustle but I can't shake the strong "accidentally swallowed the same BS we fed the VCs" vibes.

But also in the same vein nothing really is that unique, if they can make some coins this way: who are we to judge?

Re: Show HN: We built a tool that automatically generates API tests

#42

Post by thiht »

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence.

The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom

Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format.

I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them.

Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.

Re: Show HN: We built a tool that automatically generates API tests

#44

Post by ushakov »

BerislavLopac wrote: post #33

How does this compare to Schemathesis [0]? [0] https://schemathesis.readthedocs.io

Mish from Step CI here

Great question. We are aware of Schemathesis

I believe the main difference is that our solution is very configurable, unopinionated and no-code. Schemathesis doesn't offer you an easy way to configure tests, relies on OpenAPI instead and for anything more complicated requires you to use Python

Maybe you could add something else?

Re: Show HN: We built a tool that automatically generates API tests

#45

Post by ushakov »

APhoenixRises wrote: post #34

How does Step CI handle testing business logic? I don't see how anything other than input/boundary testing could be generated automatically.

Currently we are doing what is called "Synthetic monitoring"

https://en.wikipedia.org/wiki/Synthetic_monitoring

You can subscribe to our newsletter if you want to get updates once we add new features or make new tools!

Re: Show HN: We built a tool that automatically generates API tests

#46

Post by ushakov »

SeriousM wrote: post #40

Some years ago I setup postman tests and run it with cli during deployment tests. The whole hassle with postman ui just makes it really not fun work with it even for simple tasks depending on each other. I wanted to use restclient from vscode where you just define requests in a raw format but there was no usable output to process for azure devops. I will give this project a chance to get things done in a convenient w…

Thank you! We'd be happy to hear about your experience with Step CI

If you have any questions, feel free to post to our GitHub Discussions: https://github.com/stepci/stepci/discussions

Re: Show HN: We built a tool that automatically generates API tests

#47

Post by ushakov »

thiht wrote: post #42

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence. The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom Here's an exampl…

Hello, Thibaut. We appreciate the honesty!

> I prefer working with no tests than automated tests

We hear this often from our user-interviews and this is something we would like to change

I have just learned about Venom from a fellow hacker yesterday. We're more focused on Web APIs and offer more features there. The key difference is that we're not just a CLI tool, but also a library, that you can use in other applications.

Our end goal is to free you from writing tests

Re: Show HN: We built a tool that automatically generates API tests

#48

Post by Wissmueller »

thiht wrote: post #42

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence. The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom Here's an exampl…

> I have strong feelings against test generation

Thanks for your critical feedback.

Could you please tell us more about your strong opinion against test generation?

Re: Show HN: We built a tool that automatically generates API tests

#49

Post by thiht »

Wissmueller wrote: post #48
thiht wrote: post #42

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence. The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom Here's an exampl…

> I have strong feelings against test generation Thanks for your critical feedback. Could you please tell us more about your strong opinion against test generation?

> Could you please tell us more about your strong opinion against test generation?

I don't believe generated tests can test anything relevant, and to me they're just a kind of mental load. One day they break, for who knows why and you have to fix them, not knowing how they work or what they do, even doubting the value they add.

To me tests should be business focused and thoroughly thought out. Knowing that your endpoint returns a JSON with a certain format is far from enough:

- it also has to return data that makes sense, generated tests cannot give me that

- and more importantly, the API endpoints that are the most useful to test have side effects. A PUT will probably modify data in a database somewhere. A POST may trigger an asynchronous action. Inputs and outputs are worth testing thoroughly, but you have to check the side effects too. In a good test you'll check what's in db before and after your actions. That's why I love Venom, because it allows me to precisely check the state of all my system.

To be fair, I also have strong feelings against generated code in general because it's always been an impediment more than anything. I worked with JHipster for some time (a Java generator) and it's still nightmare fuel to me.

Again, take this with a grain of salt, others may have a different vision on how testing should work.

Re: Show HN: We built a tool that automatically generates API tests

#50

Post by ushakov »

barefeg wrote: post #24
Wissmueller wrote: post #23

Earlier quoted context omitted.

Thank you for the question!:) You can generate tests from your OpenAPI definition using the “import from OpenAPI” button on our website. The tests are generated from your request/response examples. If there are no examples given/available, we will generate examples for you based on the schema provided.

Thanks for the quick answer! Are the examples generated based on the types? As in if a field is set as an integer, the tests will pass arbitrary integers? But still how does it know how to produce positive examples? Do you maybe have some documentation on this functionality?

Yes! We generate placeholder values that are based on the types defined in your schema

We don't have documentation for it at the moment, but the code for our OpenAPI integration is open-source on GitHub: https://github.com/stepci/plugin-openapi

Post reply on HN