Running three hours of Ruby tests in under three minutes

#1

Post by nelhage » Thu, Aug 13, 2015, 4:25 PM UTC

Running three hours of Ruby tests in under three minutes
stripe.com

Re: Running three hours of Ruby tests in under three minutes

#2

Post by jtchang » Thu, Aug 13, 2015, 5:07 PM UTC

Love this. Sometimes testing can be a huge pain in the ass. I know more than one project I work on where getting them to run is a lot of effort in itself.

There is something to be said about code quality and having tests run in under a few seconds. The ideal situation is when you can have a barrage of tests run as fast as you are making changes to code. If we ever got to the point of instant feedback that didn't suck I'd think we'd change a lot about how we think about tests.

Re: Running three hours of Ruby tests in under three minutes

#3

Post by clayallsopp » Thu, Aug 13, 2015, 5:14 PM UTC

I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too)

My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

Re: Running three hours of Ruby tests in under three minutes

#4

Post by atonse » Thu, Aug 13, 2015, 5:18 PM UTC

On a previous project, I had built a shell script that essentially created n mysql databases and just distributed the test files under n rails processes.

We were able to run tests that took an hour in about 3 minutes. It was good enough for us. Nothing sophisticated for evenly balancing the test files, but it was pretty good for 1-2 days of work.

Re: Running three hours of Ruby tests in under three minutes

#5

Post by sytse » Thu, Aug 13, 2015, 5:25 PM UTC

Very cool stuff. For reference at GitLab we use a less impressive and simpler solution. We split the jobs off in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-... These jobs will be done by separate runners, this brought our time down from 1+ hours to 23 minutes https://ci.gitlab.com/projects/1/refs/respect_filters/commit...

Re: Running three hours of Ruby tests in under three minutes

#6

Post by sanderjd » Thu, Aug 13, 2015, 5:27 PM UTC

clayallsopp wrote: post #3

I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

I'm very curious about that as well. I worked on a big project that had a (perhaps analogous) large external dependency on networks of embedded devices in homes and businesses, and integration testing it was …difficult. I'd love to hear how Stripe solves that problem.

Re: Running three hours of Ruby tests in under three minutes

#7

Post by crdoconnor » Thu, Aug 13, 2015, 5:42 PM UTC

sanderjd wrote: post #6

clayallsopp wrote: post #3

I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

I'm very curious about that as well. I worked on a big project that had a (perhaps analogous) large external dependency on networks of embedded devices in homes and businesses, and integration testing it was …difficult. I'd love to hear how Stripe solves that problem.

Could you not create mock embedded devices?

Re: Running three hours of Ruby tests in under three minutes

#8

Post by arturhoo » Thu, Aug 13, 2015, 5:42 PM UTC

Congratulations on what look a very challenging task. I'm assuming a part of those tests hit a database. How have you dealt with it? I assume that a single instance, even on a powerful bare server could be a road blocker in this situation. A few insights on the Docker/Containerization part of it would also be nice!

Re: Running three hours of Ruby tests in under three minutes

#9

Post by nelhage » Thu, Aug 13, 2015, 5:51 PM UTC

arturhoo wrote: post #8

Congratulations on what look a very challenging task. I'm assuming a part of those tests hit a database. How have you dealt with it? I assume that a single instance, even on a powerful bare server could be a road blocker in this situation. A few insights on the Docker/Containerization part of it would also be nice!

Our testing running infrastructure spins up a pool of database instances on each worker machine, one for each worker process. The test spinup and teardown code handles schema management, hooking into our DB access layer to create and clean up database tables only if they're used by a given test.

Re: Running three hours of Ruby tests in under three minutes

#10

Post by ryanong » Thu, Aug 13, 2015, 5:53 PM UTC

If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github.

https://github.com/tmm1/test-queue

One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.