No post body was provided.
Testing in production: using JSON Schema for 3rd party API response validation
1–10 of 24 posts
Re: Testing in production: using JSON Schema for 3rd party API response validation
#2I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience..
As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
Re: Testing in production: using JSON Schema for 3rd party API response validation
#3I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience.. As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
[deleted]
Re: Testing in production: using JSON Schema for 3rd party API response validation
#4I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience.. As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
I used https://github.com/ForbesLindesay/typescript-json-validator for a service I built recently. It parses TS types and turns them into a validator function - seems to work very well so far.
There’s a conversion of the type to JSON schema under the hood but I don’t have to worry about doing it by hand.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#5I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience.. As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
I've used JSON schema a fair bit for config files for provisioning infrastructure and its been great. For example I have a kubernetes clusters JSON file cleanly schematized which specs all the clusters I need (node pools, metadata etc), which I generate TS types from using 'json-schema-to-typescript', and then just iterate over these in pulumi. All I need to do is update the JSON file to update my infra, and the coding experience is brilliant with pretty much complete type safety.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#6I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience.. As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
> As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome
Agreed. But if you want to have validation and autocompletion in most editors, it is still the way to go. We use it to validate JSON content files for a CMS [1]. It is a little repetitive in our case (e.g. the `:name`/`:description` keys are repeated), but not that bad. Newer versions of JSON Schema [2] have improved in that respect, but most editors (or plugins) and tooling in general is still based on older versions. So we have to stick with those for a while.
[1] https://github.com/frontaid/schema/blob/master/frontaid-sche...
[2] https://json-schema.org/specification.html
If you have to deal with JSON a lot, you might want to checkout https://www.schemastore.org/json/ which has schemas for a large number of file types.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#7There's a variant of this in $enterprise-dayjob: production system Foo is a service exposing a FooAPI with a spec documented in some machine readable format (SOAP ; Swagger/OpenAPI ; gRPC ). You don't need to infer the spec, you just get a copy of the spec in a machine readable format. You code generate a FooAPI client from the API spec, maybe you rig your API client to fail if it cannot decode a message with respect to the spec (it might do this by default if your code generator is any good). Maybe your new thing-with-a-FooAPI-client goes into production and you find out, lo and behold, the production FooAPI implementation doesn't honour its own spec, and encodes its data in some wacky way 5% of the time.
One stupid solution to this, that I would love to deploy, would be a circuit-breaker to detect API implementation defects. Similarly to how a circuit-breaker attempts to infer if a service is down/overloaded and temporarily disable the integration, an API-spec circuit breaker could be deployed with a copy of the API spec the external service is meant to be serving, and it could infer if that service doesn't actually implement the API spec. If it detected a counterexample from a message sent by the external service, it could disable the integration for an appropriate backoff retry interval to give the team maintaining the service enough time to align their spec or implementation (sleep 3 months to give the portfolio an opportunity to prioritise a fix; retry; sleep another three months... ).
Less stupidly, a lame but pragmatic solution can be to figure out the absolute minimal subset of the external API your use case needs in order to function. If an external service publishes an API spec with a huge service area [+], but you only need to use 1% of it, you might be advised to prune a copy of that API spec down that only focuses on the 1% you need. Validate that part of the message. Then if the external service does something weird in parts of the response you don't care about, you can ignore it instead of rejecting the entire payload.
[+] "service area" was a mistake, but it seems appropriate. maybe i need a circuit breaker between my brain and the keyboard.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#8I tried to introduce `zod` (https://github.com/colinhacks/zod) to achieve the same thing within my current team.
The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because I didn't want to expose us to the possibility that validation could take down the app.
However, TypeScript is new to my current team, and so nobody wanted the introduction of a validation API on top. It ended up being ~20 lines of code I couldn't get merged. Since TypeScript was new to the team syntax like `z.infer`, generic code and a whole new way to define types at runtime made them feel like it was too complicated.
Did others have similar problems introducing something like this?
Re: Testing in production: using JSON Schema for 3rd party API response validation
#9I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience.. As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)
I wouldn't write JSON schema definitions directly. Check out typebox. It lets you define a schema in memory and immediately have access to the types https://github.com/sinclairzx81/typebox
Combined with ajv you can even have more advanced custom validation with custom error messages.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#10I tried to introduce `zod` ( https://github.com/colinhacks/zod ) to achieve the same thing within my current team. The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because…
I struggled introducing fp-ts into my last team but io-ts was a much easier sell, as was fast-check.
It's a lot easier to get buy-in when you can demonstrate that it will reduce bugs thereby improving the customer experience and reducing development burden.
Something like io-ts or zod gives you confidence that the types in your application are correct, so you don't need to code or unit test defensively. This ties into the idea of pushing unsafety to the edges of your application. It can also be a quick way to discover that types don't align because of a backend API change; we had our io-ts decoding hooked up to Sentry.
Likewise with fast-check I argued its utility in its introduction PR and made clear that it's an opt-in method of testing, and it caught a bug we hadn't spotted within a few weeks. You can't really argue with results like that.
At a certain point though, if your team is unwilling or unable to recognise the limitations of TypeScript and therefore the necessity of constructs like `z.infer` or `t.TypeOf`, you might just have to accept it or move on. In my experience you can't move a team that far from where they'd already be heading without your presence. If you're surrounded by folks without experience in static typing, it's already a major win to have introduced TypeScript at all.