Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

docs.multiple.dev

Post reply on HN
Print view View original

1–10 of 17 posts

Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#1

Post by yevyevyev »

Hey HN,

We just shipped a new AI-powered feature... BUT the "AI" piece is largely in the background. Instead of relying on a chatbot, we've integrated AI (with strict input & output guardrails) into a workflow to handle two specific tasks that would be difficult for traditional programming:

1. Identifying the most relevant base URL from HAR files, since it would be tedious to cover every edge case or scenario to omit analytics, tracking, and other network noise.

2. Generating synthetic data for API requests by passing the API context and faker-js functions to GPT-4.

The steps are broken down into a simple flow, with users working with the AI and verifying the output throughout.

All of the focus is on reducing cognitive load and speeding up test generation.

Let me know what you think!

Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file
docs.multiple.dev

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#4

Post by yevyevyev »

xwowsersx wrote: post #3

This can be used to generate data given an openAPI spec? Bit unclear on whether that's unbundled from test generation. Say I just want to generate data that conforms to a spec as one-off. Can this be done?

That’s a great question. The TestGen feature generates JavaScript that uses faker-js functions to generate the test data, and axios to make http requests. You can copy and paste the JS output of TestGen and run it anywhere.

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#5

Post by yevyevyev »

ushakov wrote: post #2

Why is AI needed for this at all? You should take a look at Schemathesis ( https://github.com/schemathesis/schemathesis )

From my understanding, Schemathesis can generate data based on a value being a string, number, boolean, etc. It also seems fairly manual to set up and has a learning curve. Our output is JavaScript that can be run anywhere.

With our TestGen feature, the AI looks at example requests in a HAR file or Swagger examples, or it can solely rely on the name of the property. From there, it automatically generates the correct type and format of data - e.g., if a field is named "address," it generates a value that looks like an address and is formatted in the same way as examples. It wouldn't be practical to cover every potential edge case and scenario without AI.

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#7

Post by edrenova »

xwowsersx wrote: post #3

This can be used to generate data given an openAPI spec? Bit unclear on whether that's unbundled from test generation. Say I just want to generate data that conforms to a spec as one-off. Can this be done?

just to jump in here - you can do this with Neosync, an OSS synthetic data generator and orchestrator for any schema and includes relational integrity, anonymization, + more

(I'm one of the co-founders)

If you're interested (github.com/nucleuscloud/neosync)

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#8

Post by pitah1 »

Interesting feature. One key thing I found when testing is that for you to reproduce the set of steps the user went through, there are a data attribute(s) that need to remain the same. For example, after login, a request for your account information will contain an account_id number that should be the same for all other account requests. If you can't guarantee this, then I don't see how you could use this in any sort of integration tests.

Isn't it simpler to use the Open API spec then generate from there?

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#9

Post by jon_a_rey »

pitah1 wrote: post #8

Interesting feature. One key thing I found when testing is that for you to reproduce the set of steps the user went through, there are a data attribute(s) that need to remain the same. For example, after login, a request for your account information will contain an account_id number that should be the same for all other account requests. If you can't guarantee this, then I don't see how you could use this in any sort…

Hi @pitah1, thanks for the comment! I'm Jon, the guy who built the feature :)

You're spot on here. Unique identifiers within a flow shouldn't necessarily be replaced with a randomly generated string. We've attacked this problem from two different angles.

1. If an API request has a JSON response, we'll generate JS load test code for that request that begins with `const responseA = await axios[...];`. You can edit the load test to use the response data in subsequent requests.

2. We also attempt to intelligently replace UUIDs (or predictable identifiers) with a placeholder like `{fieldName}`. This highlights the values that need user intervention.

We use the Swagger schema to determine the available API endpoints, if each endpoint has a request body vs query string params vs none, and so on. We sprinkle in AI to help decide how to best saturate request bodies with realistic data via faker.js.

Re: Show HN: Auto-generate load tests/synthetic test data from OpenAPI spec/HAR file

#10

Post by yevyevyev »

cebert wrote: post #6

How is this different from Postman’s test generation features? https://www.postman.com/postman-galaxy/dynamically-generate-...

Postman generates data based on datatypes in the OpenAPI spec: strings, numbers, booleans, etc - but the data will not look realistic. The video outlined a rudimentary test that checked if required fields were present.

Our TestGen feature generates realistic-looking data, such as dates, names, addresses, URLs, etc, automatically based on the field names, examples, and other API spec metadata. It does this automatically, without human intervention. The output is JavaScript, so if further customization is needed, such as using a response value of an API call in a subsequent request, you can do that.

Post reply on HN