Json schema validation is cpu expensive in nodejs, so you do not want it in production for every request. We made a proxy [1] that only allows requests that are conform to the swagger json which is easy to hook between frontend and backend in dev-mode
Testing in production: using JSON Schema for 3rd party API response validation
11–20 of 24 posts
Re: Testing in production: using JSON Schema for 3rd party API response validation
#12As someone who's about to assist a frontend team to implement testing against a backend I've developed (my first time doing something like this), I can see that mocking backend APIs is inherently tedious and error-prone. And I'm not confident that we have the resources to keep the mocked endpoints consistent with the real backend given the pressure to develop the "real product".
What are the downsides of of using a real API (e.g. dev environment) for automated testing?
Re: Testing in production: using JSON Schema for 3rd party API response validation
#13Back in the early years of Snowplow, we adopted a similar approach for our support of 3rd party webhooks[1] as an event stream source supported natively by Snowplow (https://github.com/snowplow/snowplow). (The important background here is that all events and entities flowing through Snowplow are described using versioned JSON Schema).
We built a kind of Maven Central for 3rd party webhooks' schemas[2], and got to work adding schemas for various popular webhooks, e.g. Sendgrid and Pingdom. But, we never met a single SaaS vendor who was interested in 'adopting' the JSON Schemas for their webhook, let alone publishing versioned JSON Schemas for their whole API. This meant that at Snowplow we have stayed on the hook for keeping these vendors' webhook schema definitions up to date in our registry.
It's a real tragedy of the commons - oceans of developer time wasted on tedious low-leverage work (updating API client code) so that each SaaS vendor can 'move fast and break things'. The ultimate irony is that if these vendors were to adopt, publish and respect versioned schema definitions internally, using something like OpenAPI, they would see huge productivity gains (think enhanced CI/CD testing, auto-gen of client code etc).
1. https://docs.snowplowanalytics.com/docs/getting-started-on-s... 2. https://github.com/snowplow/iglu-central/tree/master/schemas
Re: Testing in production: using JSON Schema for 3rd party API response validation
#14As someone who's about to assist a frontend team to implement testing against a backend I've developed (my first time doing something like this), I can see that mocking backend APIs is inherently tedious and error-prone. And I'm not confident that we have the resources to keep the mocked endpoints consistent with the real backend given the pressure to develop the "real product". What are the downsides of of using a r…
> What are the downsides of of using a real API (e.g. dev
> environment) for automated testing?
Re: Testing in production: using JSON Schema for 3rd party API response validation
#15I tried to introduce `zod` ( https://github.com/colinhacks/zod ) to achieve the same thing within my current team. The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because…
> and a whole new way to define types at runtime made them feel like it was too complicated
How you made this happen? My impression was that TypeScript doesn't have runtime types and stopped looking at TypeScript after that as it basically makes TypeScript useless for myself and my team.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#16 Consider adding "additionalProperties": false as you tweak the JSON Schema. That way, you will also get alerts as new fields are added. https://json-schema.org/understanding-json-schema/reference/...
Re: Testing in production: using JSON Schema for 3rd party API response validation
#17As someone who's about to assist a frontend team to implement testing against a backend I've developed (my first time doing something like this), I can see that mocking backend APIs is inherently tedious and error-prone. And I'm not confident that we have the resources to keep the mocked endpoints consistent with the real backend given the pressure to develop the "real product". What are the downsides of of using a r…
> What are the downsides of of using a real API (e.g. dev environment) for automated testing?
Slower, harder to set up state the tests need.
Front end teams should also be testing their code against a broken back end; eg http 500 responses.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#18I tried to introduce `zod` ( https://github.com/colinhacks/zod ) to achieve the same thing within my current team. The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because…
> and a whole new way to define types at runtime made them feel like it was too complicated How you made this happen? My impression was that TypeScript doesn't have runtime types and stopped looking at TypeScript after that as it basically makes TypeScript useless for myself and my team.
Libraries like io-ts and zod take care of this for you. For io-ts for example you just need to define a type guard function that asserts "x is Y" and away you go, you've got a runtime codec; it knows how to "decode" data, and it knows what that represents to the type system at compile-time. You'll generally only need to use the built-in combinators though e.g. `t.type({ prop: t.string })` produces a codec for `{ prop: string }`, and you can get that type at compile-time without effectively rewriting it using `t.TypeOf`.
Re: Testing in production: using JSON Schema for 3rd party API response validation
#19Earlier quoted context omitted.
> and a whole new way to define types at runtime made them feel like it was too complicated How you made this happen? My impression was that TypeScript doesn't have runtime types and stopped looking at TypeScript after that as it basically makes TypeScript useless for myself and my team.
Libraries like io-ts and zod take care of this for you. For io-ts for example you just need to define a type guard function that asserts "x is Y" and away you go, you've got a runtime codec; it knows how to "decode" data, and it knows what that represents to the type system at compile-time. You'll generally only need to use the built-in combinators though e.g. `t.type({ prop: t.string })` produces a codec for `{ prop…
It's a nice library, although I personally prefer explicitly writing the type out beside the codec. Otherwise you have to translate the codec to the type in your head when reading the code without an ide that can show you the type.
This also gives you a "test" if you will that the codec is parsing the structure you think it is
Re: Testing in production: using JSON Schema for 3rd party API response validation
#20I tried to introduce `zod` ( https://github.com/colinhacks/zod ) to achieve the same thing within my current team. The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because…
Yes definitely. I have trouble just introducing Python type hints, despite the clear insanity of working with a large convoluted untyped codebase.