Divide and Concur - testing code at etsy • Hacker News Board

Divide and Concur - testing code at etsy

#1

Post by dajobe » Sun, May 01, 2011, 4:22 PM UTC

Divide and Concur - testing code at etsy
codeascraft.etsy.com

Re: Divide and Concur - testing code at etsy

#2

Post by bestes » Sun, May 01, 2011, 7:39 PM UTC

Sounds like you've developed a great system/process here. Are you a developer or QA? Are there two groups or even just a person with testing as their responsibility?

Have you ever had people mark tests @flaky just so they could deploy (and that later caused grief)?

I would love to hear more.

Re: Divide and Concur - testing code at etsy

#3

Post by dajobe » Sun, May 01, 2011, 11:26 PM UTC

bestes wrote: post #2

Sounds like you've developed a great system/process here. Are you a developer or QA? Are there two groups or even just a person with testing as their responsibility? Have you ever had people mark tests @flaky just so they could deploy (and that later caused grief)? I would love to hear more.

I submitted the blog post, I am not the author of it.

Re: Divide and Concur - testing code at etsy

#4

Post by NoahSussman » Sun, May 01, 2011, 11:35 PM UTC

bestes wrote: post #2

Sounds like you've developed a great system/process here. Are you a developer or QA? Are there two groups or even just a person with testing as their responsibility? Have you ever had people mark tests @flaky just so they could deploy (and that later caused grief)? I would love to hear more.

Both of the authors are Web developers with a deep interest in software quality and automated testing. Etsy has followed the same software quality model as Facebook: a dedicated testing team, but no dedicated manual QA team. See also http://b.qr.ae/hJVb4W

Etsy's engineering culture puts great emphasis on personal responsibility and taking pride in the quality of one's work. Because of that, no, there has never yet been a problem with people taking tests offline simply so they could deploy. The issue is more that tests "wear out" over time and need to be either rewritten, replaced or discarded (for more on this phenomenon see Boris Beizer and his "pesticide paradox").

Re: Divide and Concur - testing code at etsy

#5

Post by draz » Mon, May 02, 2011, 2:21 AM UTC

sounds cool, though, with 25 deployments a day, it seems like you're just deploying quick fixes. How involved could changes be with such small time intervals? I applaud the approach though. Also, could you also talk about how you track all the end-user impact, given the very frequent changes? How does this feed back into the analytics (is change A causing rise/fall in conversion, or change B)?

Re: Divide and Concur - testing code at etsy

#6

Post by NoahSussman » Mon, May 02, 2011, 10:17 AM UTC

draz wrote: post #5

sounds cool, though, with 25 deployments a day, it seems like you're just deploying quick fixes. How involved could changes be with such small time intervals? I applaud the approach though. Also, could you also talk about how you track all the end-user impact, given the very frequent changes? How does this feed back into the analytics (is change A causing rise/fall in conversion, or change B)?

Yes, most deployments contain only a couple of changesets -- ideally no more than a few dozen lines of code per push. And even these small changes are deployed behind config flags. Flickr has a similar continuous deploy strategy, see http://bit.ly/esoXiX

There is intended to be no end user impact whatsoever. Releasing behind config flags means that feature rollouts are decoupled from deploying code. New code runs "dark" for a long time in production. There we can log and graph and adjust its performance before it is ever visible to the user.

And of course, we can see dark features in production before they're released or even complete. This means that at Etsy we've been using and tweaking our features long before they are made public.

Since we've been closely monitoring and recording the performance of each new feature since its inception, we are highly aware of how that changes once the code is live to a known, small number of our users. Gather data, make adjustments, rinse and repeat until eventually the feature is live to 100% of users.