Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polly.js – Record, replay, and stub HTTP interactions

netflix.github.io

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Polly.js – Record, replay, and stub HTTP interactions

#81

Post by ryanbrunner »

fzaninotto wrote: post #63
stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

Also, recording raw HTTP requests makes tests difficult to organize. Especially if what you want to mock are requests to a REST server. In that case, all the recorded HTTP headers aren't significant, editing the recorded resources in the responses when the API changes is a pain, and testing scenarios where several REST requests are related (e.g. fetching posts than comments) is also a pain. A better alternative IMO i…

Insignificant things like HTTP headers and extra fields are insignificant until they're not. In my experience, manually assembling what you expect an HTTP response to look like often leads to bugs when an "irrelevant" detail suddenly becomes relevant, like when status codes change, or fields are added in a way that breaks a client, etc.

I think recording tools can be a sharp tool, and require care, but (as a starting point), if you have an automated library that can generate recorded fixtures in a repeatable, automated fashion, you can eliminate a lot of the pain points while still reaping all of the benefits. That's how we set it up where I am - responses and fixtures are generated as part of a full suite execution, but persist with individual test runs.

Re: Polly.js – Record, replay, and stub HTTP interactions

#82

Post by emodendroket »

stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

I worked on a Ruby project with thousands of VCR recordings. Never again.

Re: Polly.js – Record, replay, and stub HTTP interactions

#83

Post by simplify »

jasonmit wrote: post #57
curun1r wrote: post #42

Earlier quoted context omitted.

Not sure that tells me how it's different from replay or the other half dozen npm modules that do the same thing. It'd be nice for them to contrast their tool with existing ecosystem options considering some of them are pretty well established.

Can you share which libraries you know that achieve the same thing? I'm happy to go through and respond to the differences.

I've personally used node-replay with great success. It has minimal configuration https://github.com/assaf/node-replay

Re: Polly.js – Record, replay, and stub HTTP interactions

#84

Post by stephen »

ryanbrunner wrote: post #80
stephen wrote: post #69

Earlier quoted context omitted.

> Again, nothing manual Yes, sorry for being inexact/overusing the term--I understand the tests drive the recording. What I meant by manual is getting the e2e system into your test's initial state. E.g. tests are invariably "world looks like X", "system under test does Y", "world looks like Z". In record/replay, "world looks like X" is not coded, isolated, documented in your test, and is instead implicit in "whatever…

But in the case we're talking about, where you're reliant on an external service that can change underneath you, "world looks like X" is genuinely not under your control. It feels like pretending that it is will lead to just as many failures as acknowledging it's inherent volatility.

Agreed! And, to me, record/replay is still pretending like it's controllable, b/c even if you decouple for replays, records will always be a PITA.

My depressing solution is to just not even try to automate tests against the upstream system and instead invest in test builders/DSLs that make mocks/stubs on both sides as pleasant as possible.

And when bugs slip through, make sure to update your stubs/mocks on both sides to prevent the regression.

To me this gets the most agility and reliability, and will be a test suite that developers don't hate 1-2-5 years down the road.

Re: Polly.js – Record, replay, and stub HTTP interactions

#85

Post by wesleytodd »

MaxLeiter wrote: post #48
error54 wrote: post #35

Earlier quoted context omitted.

Not to shamelessly plug but if you're in the Bay Area on June 28, we're giving a talk that's a bit about performance, a bit about Netflix engineering culture: https://jstalks2018.splashthat.com/ .

If I’m a student can I sign up to attend?

Sure! (at least I don't see why you wouldn't be able to)

Re: Polly.js – Record, replay, and stub HTTP interactions

#86

Post by mikekchar »

stephen wrote: post #70
mikekchar wrote: post #68

Earlier quoted context omitted.

I also don't like recording frameworks for TDD (and similarly dislike using fixtures). However, the place where a recording framework really pays for itself is in isolating changes in protocols. I've often had to interface with, as you put it, uncontrollable upstream systems. These are systems that are not mine -- they are upstream services from other companies that I have to interact with and I have no control over.…

> isolating changes in protocols That makes sense. Ideally protocols are declarative/documented/typed, e.g. Swagger/GRPC, so you can be more trusting and not need these, but often in REST+JSON that does not happen. > All the rest of my code is coded against the adaptor Nice, I like (the majority?) of tests being isolated via that abstraction. Although, if "the protocol" is basically "all of the JSON API calls my weba…

Yes, I like to have scenarios for all of the end points. It definitely doesn't reduce the number of tests :-) The advantage is in isolating the protocol from the operation of the application.

The main complaint I've seen for this approach is, "We shouldn't be testing the other person's system". There is wisdom in that advice, but it really depends on how much you depend on the 3rd party service and how much downtime you can tolerate. For example, I'm working in the travel industry right now and we often rely on small services that nobody has ever heard of. If we can't use the service then we can't sell anything and our site is essentially down. If it happens frequently (and with a lot of these travel services, they often break things weekly if not daily), then your site is not viable. In that case I'll exercise the protocol as much as I can. However, we also talk to marketing services, etc. If that breaks, and it takes a day or two to get it back up, then it's not a major problem -- our marketing effort might be a day late, which is unfortunate, but not game breaking. In that case I'll usually have a smoke test or two.

Re: Polly.js – Record, replay, and stub HTTP interactions

#87

Post by Mandatum »

reymus wrote: post #33
Mandatum wrote: post #25

Earlier quoted context omitted.

This won't work for WebSockets, really websites that use WebSockets require some interaction to generate the transmitted message which is often dependent on the servers response. Private websites, or websites that require a login are hard - but it can be done. Would suggest HTTrack.

But it's not impossible to have some tool that records all of those interactions to reproduce later. A smart enough tool could record everything since you open the site until you click save. It would not reproduce the functionality that is backend dependent, but iy sure can replicate the dom, etc. Am I missing something?

Yeah - but there's a LOT of variables that come into play for something like that. It'd likely be easier to either record it with something like BugReplay.com or video.

Re: Polly.js – Record, replay, and stub HTTP interactions

#89

Post by raywu »

jasonmit wrote: post #53
stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

> you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) Your test invokes the the recorder. There isn't anything manual outside of writing & running your test. > you have to manually re-record when they fail Again, nothing manual. It would require running your test again with Polly in record mode if you want to "…

Can you post the tutorial here? Thanks @jasonmit

Re: Polly.js – Record, replay, and stub HTTP interactions

#90

Post by jmckenz »

ascendantlogic wrote: post #78

I'm curious what the application could be for load testing? Tools like locust and gatling are nice but are still synthetic. I'd love to capture X minutes of traffic, then dupe it Y times and replay it as a more accurate representation of traffic patterns for load testing. Is that a thing?

not had a chance to properly try yet, but https://goreplay.org/ does exactly what you are asking. Alternatively, in the container world, tools such as Istio (https://istio.io/) allow traffic shadowing - you can duplicate traffic and route it somewhere else
Post reply on HN