Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running three hours of Ruby tests in under three minutes

stripe.com

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Running three hours of Ruby tests in under three minutes

#21

Post by pekk »

teacup50 wrote: post #15

How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…

Thread-based concurrency based on shared mutable state doesn't reduce complexity.

Re: Running three hours of Ruby tests in under three minutes

#22

Post by dankohn1 »

We're not nearly at Stripe's scale, but my startup (Spreemo) has achieved pretty amazing parallelism using the commercial SaaS CircleCI. We have 3907 expects across 372 RSpec and Cucumber files. Our tests complete in ~14 minutes when run across 8 containers.

One of the great strengths for CircleCI is that they auto-discover our test types, calculate how long each file takes to run, and then auto-allocate the files in future runs to try to equalize the run times across containers. The only effort we had to do was split up our slowest test file when we found that it was taking longer to complete than a combination of files on the other machines.

I also like that I can run pronto https://github.com/mmozuras/pronto to post Rubocop, Rails Best Practices, and Brakeman errors as comments on Github.

Re: Running three hours of Ruby tests in under three minutes

#23

Post by andreasklinger »

dankohn1 wrote: post #22

We're not nearly at Stripe's scale, but my startup (Spreemo) has achieved pretty amazing parallelism using the commercial SaaS CircleCI. We have 3907 expects across 372 RSpec and Cucumber files. Our tests complete in ~14 minutes when run across 8 containers. One of the great strengths for CircleCI is that they auto-discover our test types, calculate how long each file takes to run, and then auto-allocate the files in…

i like pronto's approach

we simply added the linters/code analysis to the CI itself

reasoning: we try to have as little as possible "code style" discussion in PRs

Re: Running three hours of Ruby tests in under three minutes

#25

Post by andreasklinger »

clayallsopp wrote: post #3

I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

Not a stripe member but i would assume that anything that involves intense security auditing, PCI, etc would be seperate codebases that rarely change.

(eg cc handling could anon the CCs in a service before they reach the main app)

The integration with 3rd parties is a seperate issue that exists no matter of it is banks or not - i would guess they abstracted that as well as services or libs and decide case by case.

Re: Running three hours of Ruby tests in under three minutes

#26

Post by praxulus »

vkjv wrote: post #14

"This second round of forking provides a layer of isolation between tests: If a test makes changes to global state, running the test inside a throwaway process will clean everything up once that process exits." But, then how do you catch bugs where shared mutable state is not compatible with multiple changes?

You write tests specifically for testing multiple changes. You shouldn't be testing changes to global state by seeing how multiple supposedly independent tests interact.

Re: Running three hours of Ruby tests in under three minutes

#27

Post by patio11 »

chinathrow wrote: post #18

Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?

how do you trust the hosted CI company not to look at your code?

Contracts, not firewalls, make the world go round.

Re: Running three hours of Ruby tests in under three minutes

#28

Post by brown9-2 »

chinathrow wrote: post #18

Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?

how do you trust the hosted CI company not to look at your code

One can probably assume that they are not relying upon the secrecy of their code for security.

Re: Running three hours of Ruby tests in under three minutes

#30

Post by sanderjd »

crdoconnor wrote: post #7
sanderjd wrote: post #6

Earlier quoted context omitted.

I'm very curious about that as well. I worked on a big project that had a (perhaps analogous) large external dependency on networks of embedded devices in homes and businesses, and integration testing it was …difficult. I'd love to hear how Stripe solves that problem.

Could you not create mock embedded devices?

That's basically what we did, but more like mocking things at the network communication boundary. But for an integration test, it was often unsatisfactory, because of things behaving differently than they do in the real world. We also had a suite of automated tests that communicated with real devices in a lab, which were much better, but extremely hard to maintain. My general experience was: tenable, but tricky. So I'd love to hear how others have handled similar things.
Post reply on HN