What would a use case of this library look like?
Polly.js – Record, replay, and stub HTTP interactions
61–70 of 91 posts
Re: Polly.js – Record, replay, and stub HTTP interactions
#62This is very cool, solving some issues I no doubt many people have when writing tests against a (fast-)moving target. I'll definitely give it a try in my next project.
I looked through the codebase, and noticed that this uses a custom data format to persist HTTP requests and responses in local storage. I'm not sure if it's technically possible in all circumstances, but I think it might be valuable to have requests and responses be stored as HAR 1.2 [1] when possible, so that the trace can be used by other tools [2] to aid in debugging, verifying and analyzing behaviour as well as perhaps automated creation of load/performance tests.
[1] - http://www.softwareishard.com/blog/har-12-spec/
[2] - e.g. https://toolbox.googleapps.com/apps/har_analyzer/
Re: Polly.js – Record, replay, and stub HTTP interactions
#63Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…
Also, recording raw HTTP requests makes tests difficult to organize. Especially if what you want to mock are requests to a REST server. In that case, all the recorded HTTP headers aren't significant, editing the recorded resources in the responses when the API changes is a pain, and testing scenarios where several REST requests are related (e.g. fetching posts than comments) is also a pain.
A better alternative IMO is to craft a list of resources in JSON, then use this data in a fake REST server that takes over fetch and XHR in the browser.
Something like:
{ posts: [{ id: 1, title: "foo" }, { id: 2, title: "bar"}], comments: [{ id: 1, post_id: 1, body : "lorem ipsum" }] }
Re: Polly.js – Record, replay, and stub HTTP interactions
#64I used to use nock which would work very well in node environments. But this works in the browser as well. So I guess this can be fairly helpful while writing tests post development. If you are doing TDD, then recording/replaying doesn't fit anywhere in the development cycle.
I like the API of this library and the browser support that was missing in nock. So thanks Netflix! Although it would have been nice to see nock add this support. Which is what I wonder - why not just contribute to existing libraries.
Re: Polly.js – Record, replay, and stub HTTP interactions
#65Also, what's up with the in-your-face hiring pitch right in the documentation?
Re: Polly.js – Record, replay, and stub HTTP interactions
#66I personally replace all stubbed HTTP interactions between my services with contracts with good success. https://docs.pact.io/getting-started
Re: Polly.js – Record, replay, and stub HTTP interactions
#67I used to use nock which would work very well in node environments. But this works in the browser as well. So I guess this can be fairly helpful while writing tests post development. If you are doing TDD, then recording/replaying doesn't fit anywhere in the development cycle. I like the API of this library and the browser support that was missing in nock. So thanks Netflix! Although it would have been nice to see noc…
If you're looking for Nock but not just node, try Mockttp: https://github.com/pimterry/mockttp.
It lets you create & configure mock HTTP servers for JS testing, but with one API that works out of the box in Node and in browsers. This avoids the record/replay model too, so you can still take the TDD approach and define your API behaviour in the test itself.
(Disclaimer: I'm the author of Mockttp)
Re: Polly.js – Record, replay, and stub HTTP interactions
#68Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…
I also don't like recording frameworks for TDD (and similarly dislike using fixtures). However, the place where a recording framework really pays for itself is in isolating changes in protocols. I've often had to interface with, as you put it, uncontrollable upstream systems. These are systems that are not mine -- they are upstream services from other companies that I have to interact with and I have no control over. Often these systems are badly built and they play fast and loose with the "protocols".
In these case I like to have an adaptor layer and use a recording framework to "test" the adaptor. That way I can occasionally rerecord my scenarios and be notified if something important has changed. Normally what happens is that my service stops working for some unknown reason. I rerecord the adaptor scenarios and usually the reason pops out very quickly. All the rest of my code is coded against the adaptor and I stub it out in their tests (which I can do reasonably well because I control it).
Re: Polly.js – Record, replay, and stub HTTP interactions
#69Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…
> you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) Your test invokes the the recorder. There isn't anything manual outside of writing & running your test. > you have to manually re-record when they fail Again, nothing manual. It would require running your test again with Polly in record mode if you want to "…
> Again, nothing manual
Yes, sorry for being inexact/overusing the term--I understand the tests drive the recording.
What I meant by manual is getting the e2e system into your test's initial state.
E.g. tests are invariably "world looks like X", "system under test does Y", "world looks like Z".
In record/replay, "world looks like X" is not coded, isolated, documented in your test, and is instead implicit in "whatever the upstream system looked like when I hit record".
Which is almost always "the developer manually clicked around a test account to make it look like X".
This is basically a giant global variable that will change, and come back to haunt you when recordings fail, b/c you have to a) re-divine what "world looks like X" was for this test, and then b) manually restore the upstream system to that state.
If no one has touched the upstream test data for this specific test case, you're good, but when you get into ~10s/100s of test, it's tempting to share test accounts, someone accidentally changes it, or else you're testing mutations and your test explicitly changes it (so need to undo the mutation to re-record), or you wrote the test 2 years ago and the upstream system aged off your data.
All of these lead to manually clicking around to re-setup "world looks like X", so yes, that is what I should have limited the "manual" term to.
Re: Polly.js – Record, replay, and stub HTTP interactions
#70Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…
I also don't like recording frameworks for TDD (and similarly dislike using fixtures). However, the place where a recording framework really pays for itself is in isolating changes in protocols. I've often had to interface with, as you put it, uncontrollable upstream systems. These are systems that are not mine -- they are upstream services from other companies that I have to interact with and I have no control over.…
> isolating changes in protocols
That makes sense.
Ideally protocols are declarative/documented/typed, e.g. Swagger/GRPC, so you can be more trusting and not need these, but often in REST+JSON that does not happen.
> All the rest of my code is coded against the adaptor
Nice, I like (the majority?) of tests being isolated via that abstraction.
Although, if "the protocol" is basically "all of the JSON API calls my webapp makes to the backend REST services", at that point do you end up adapter scenarios (record/replay recordings) for basically every use case anyway? Or do you limit it to only a few endpoints or only a few primary operations?