Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polly.js – Record, replay, and stub HTTP interactions

netflix.github.io

Post reply on HN
Print view View original

71–80 of 91 posts

Re: Polly.js – Record, replay, and stub HTTP interactions

#71

Post by stephen »

jasonmit wrote: post #53
stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

> you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) Your test invokes the the recorder. There isn't anything manual outside of writing & running your test. > you have to manually re-record when they fail Again, nothing manual. It would require running your test again with Polly in record mode if you want to "…

> It's a tool for your toolbox, reach for it when needed

Sure, apologies for being negative about a tool you've worked on and are rightly proud of. I'm sure you already have more users than any open source project I've ever written. :-)

I struggle a bit at this point in my career, as I've made enough mistakes and seen enough mistakes, that I generally have strong gut opinions on "yeah, that's probably not going to work/scale/etc."

So, when observing new developers/teams starting to "make a mistake" that I've seen before, my gut says "no! bad idea!"...but I know I could be wrong, so it's tempting to say "well, sure, that didn't work for us, but go ahead and try again".

Because, who knows, maybe eventually someone will figure out an innovation that makes a previously-bad approach now tenable, and even best-practice.

But, realistically, that rarely happens, and so teams, orgs, the industry as a whole stumbles around re-making the same mistakes, and codebases/teams/etc. pay the cost.

I've thought a lot about micro-service testing at scale:

http://www.draconianoverlord.com/2018/01/21/microserving-tes...

Basically there are no easy answers, short of some sort of huge, magical, up-front investment in testing infra that only someone like a top-5/top-10 tech company has the eng resources to do.

So, definitely appreciate needing to do "something else" in the mean time. ...record/replay is just not a "something else" I would go with. :-)

Re: Polly.js – Record, replay, and stub HTTP interactions

#72

Post by xab9 »

I did something similar, but as an interim proxy (can record, replay, there are modifier hooks, can slow down requests). You have to point towards a backend api and on the frontend you use the proxy url instead of the original. But it's mostly for debugging, so the scope is much more limited.

Re: Polly.js – Record, replay, and stub HTTP interactions

#73

Post by thijsvandien »

dudul wrote: post #19
mygo wrote: post #8

I know this is slightly different, but I wish more people knew about Chrome / Safari / Firefox’s “network” console tab. Great for debugging. Can look at all requests, headers, responses, timespans, etc. Some will even let you copy a given network request as a cURL command, capturing all headers, body, query strings, etc.

And out of curiosity, what makes you think that people don't know about it? I've never met a web dev who didn't know about it in the past few years.

New people are introduced to web development every day. Assuming some things are just common knowledge is not very beginner friendly. https://xkcd.com/1053/

Re: Polly.js – Record, replay, and stub HTTP interactions

#74

Post by m12k »

mikekchar wrote: post #68
stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

I also don't like recording frameworks for TDD (and similarly dislike using fixtures). However, the place where a recording framework really pays for itself is in isolating changes in protocols. I've often had to interface with, as you put it, uncontrollable upstream systems. These are systems that are not mine -- they are upstream services from other companies that I have to interact with and I have no control over.…

I've worked for a while with similar scenarios of needing to integrate with systems beyond my control (e.g. Stripe, Slack, Google), and though I still don't have a good setup for it, I've come to the conclusion that a two-pronged approach would be ideal: Record/and/or stub the calls and responses from the external services so your normal tests are run entirely without external networking (like what Polly allows you to do). But also set up a server to periodically validate the responses from the external services against those that have been recorded, and alert you to any changes in their protocol/behavior. I've yet to see any middleware to tackle the latter (though granted, I haven't been looking too hard yet either)

Re: Polly.js – Record, replay, and stub HTTP interactions

#75

Post by anchpop »

vaidik wrote: post #65

Also, what's up with the in-your-face hiring pitch right in the documentation? https://netflix.github.io/pollyjs/#/README?id=we39re-hiring

I assume that devs read the documentation, and they want to hire devs, and it's their tool, so they put their hiring pitch in their documentation for their tool to try and hire devs

Re: Polly.js – Record, replay, and stub HTTP interactions

#76

Post by mbell »

stephen wrote: post #46

Looks well done (it uses unicode art, so it must be amazing) but I have a fundamental distrust/dislike of record/replay frameworks...just seems like you're papering over an inherently bad testing approach. E.g. sure, when replays work, they're great, but: a) you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) b…

Not sure I understand what you are proposing as an alternative. It seems that you can either test against a 'real' system on the other end, which probably means one project pulling, building and standing up possibly dozens of other services just to run it's test suite. Or, you mock it in some way. I prefer the recording approach as it mocks at the lowest level possible giving you the most test coverage possible.

Re: Polly.js – Record, replay, and stub HTTP interactions

#77

Post by aspencer8111 »

drb91 wrote: post #58
jasonmit wrote: post #55

Earlier quoted context omitted.

Polly records as well as exposes a stubbing API. So it's quite different from what I've seen of the others.

Ahh. Well, there’s quite a history of recording as well as stubbing: see ruby’s VCR.

Came here say the exact same thing: "Hey, look! Its VCR for JS. Yay!"

Re: Polly.js – Record, replay, and stub HTTP interactions

#78

Post by ascendantlogic »

I'm curious what the application could be for load testing? Tools like locust and gatling are nice but are still synthetic. I'd love to capture X minutes of traffic, then dupe it Y times and replay it as a more accurate representation of traffic patterns for load testing. Is that a thing?

Re: Polly.js – Record, replay, and stub HTTP interactions

#79

Post by lhorie »

burger_moon wrote: post #10

I would love to hear from people involved in projects like this, what kind of work/ how much, was done to get it ready for and approved to be open sourced by the company. Especially at large corps like Netflix I'm sure there's a lot of hoops to jump through.

In my company (uber), it's actually not a whole lot of hoops. Basically a light legal review that checks the license, a code review to ensure there aren't references to closed source software and infrastructure, and approval from the team manager, who is usually already on board with the desire to open source.

Re: Polly.js – Record, replay, and stub HTTP interactions

#80

Post by ryanbrunner »

stephen wrote: post #69
jasonmit wrote: post #53

Earlier quoted context omitted.

> you have to do a manual recording to create them the first time (which means manually setting up test data in production that is just-right for your test case) Your test invokes the the recorder. There isn't anything manual outside of writing & running your test. > you have to manually re-record when they fail Again, nothing manual. It would require running your test again with Polly in record mode if you want to "…

> Again, nothing manual Yes, sorry for being inexact/overusing the term--I understand the tests drive the recording. What I meant by manual is getting the e2e system into your test's initial state. E.g. tests are invariably "world looks like X", "system under test does Y", "world looks like Z". In record/replay, "world looks like X" is not coded, isolated, documented in your test, and is instead implicit in "whatever…

But in the case we're talking about, where you're reliant on an external service that can change underneath you, "world looks like X" is genuinely not under your control. It feels like pretending that it is will lead to just as many failures as acknowledging it's inherent volatility.
Post reply on HN