Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running three hours of Ruby tests in under three minutes

stripe.com

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Running three hours of Ruby tests in under three minutes

#81

Post by givehimagun »

cthyon wrote: post #80

Not sure if this has already been answered, but would Stripe's methods only work with unit tests where tests are not dependent on each other? How would one go about building a similar distributed testing setup for end-to-end tests where a sequence of tests have to be run in particular order. Finding the optimal ordering / distribution of tests between workloads would certainly be more complicated. Maybe they could be…

I'd love to know if their integration tests use a database or reference external services of any sort.

We ended up making a compromise where each test can never expect another test to have run...but some tests expect certain test data to be present and in a known state. To handle that, every test cleans up the data of the previous run (Entity Framework has a nice change tracker where we can keep track of the unit of work before it is persisted). We wouldn't be able to parallelize everything though...we can only accept a single test to be active on the DB at a single point in time.

Re: Running three hours of Ruby tests in under three minutes

#82

Post by matthewmacleod »

cthyon wrote: post #80

Not sure if this has already been answered, but would Stripe's methods only work with unit tests where tests are not dependent on each other? How would one go about building a similar distributed testing setup for end-to-end tests where a sequence of tests have to be run in particular order. Finding the optimal ordering / distribution of tests between workloads would certainly be more complicated. Maybe they could be…

How would one go about building a similar distributed testing setup for end-to-end tests where a sequence of tests have to be run in particular order.

I reckon that would be solving the wrong problem. End-to-end tests should be independent of each other, and tests should never be dependent on the order in which they are run. End-to-end tests might be longer as a result, but managing the complexity of test dependencies will quickly cripple any system that uses this approach, I imagine.

Re: Running three hours of Ruby tests in under three minutes

#83

Post by notduncansmith »

cthyon wrote: post #80

Not sure if this has already been answered, but would Stripe's methods only work with unit tests where tests are not dependent on each other? How would one go about building a similar distributed testing setup for end-to-end tests where a sequence of tests have to be run in particular order. Finding the optimal ordering / distribution of tests between workloads would certainly be more complicated. Maybe they could be…

I think those would not be considered "unit" tests. Often the definition of unit tests includes the ability to run those tests in any order. Any tests that have to be run in a particular order (i.e. "stateful" tests) should be considered a single test, and likely an integration test at that.

Re: Running three hours of Ruby tests in under three minutes

#84

Post by amalag »

teacup50 wrote: post #15

How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…

JRuby will do the first and part of the second

Re: Running three hours of Ruby tests in under three minutes

#85

Post by speedkills »

com2kid wrote: post #60

I am tired of this technology having to be re-invented time and time again. The best I ever saw was an internal tool at Microsoft. It could run tests on devices (Windows Mobile phones, but it really didn't care), had a nice reservation and pool system and a nice USB-->Ethernet-->USB system that let you route any device to any of the test benches. This was great because it was a heterogeneous pool of devices, with dif…

I agree. We already have projects like http://test-load-balancer.github.io but I have a feeling I will see five more posts on he in the next year about re-inventing this wheel and yet not see a single contribution to existing solutions like tlb.

It must be a little depressing to build a really useful product you know many people need, give it away only hoping people will use it and be happy, then find out everyone would rather build their own.

But we do like to build things, it is in our nature. Plus, what looks better on your resume: 1) I migrated my teams test suite to using test load balancer in two days, saving hours every test run. 2) I contributed improvements to the open source test load balancer project. 3) I designed and implemented my own distributed test load balancing tool!

Re: Running three hours of Ruby tests in under three minutes

#86

Post by hawkice »

brown9-2 wrote: post #28
chinathrow wrote: post #18

Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?

how do you trust the hosted CI company not to look at your code One can probably assume that they are not relying upon the secrecy of their code for security.

There are other reasons to keep code proprietary than fearing a security failure in the event the code leaks.

Re: Running three hours of Ruby tests in under three minutes

#88

Post by jwatte »

chinathrow wrote: post #18

Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?

If their code is right, everyone in the world reading it wouldn't be a problem.

Re: Running three hours of Ruby tests in under three minutes

#90

Post by akoumjian »

falsedan wrote: post #35

Oh hey, we have the same sort of system here. It's 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI & scheduler & artifact distribution system (which we're probably going to replace with S3). We run selenium & unit tests as well as the integration tests. We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dock…

Anything significantly different in your Python implementation?
Post reply on HN