It's interesting to imagine, for a test suite that would take three hours, how much of the execution time is state management vs algorithm execution.
Running three hours of Ruby tests in under three minutes
11–20 of 111 posts
Re: Running three hours of Ruby tests in under three minutes
#12If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github. https://github.com/tmm1/test-queue One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.
test-queue supports minitest too, and follows the same basic design outlined in this article: a central queue sorted by slowest tests first, with test runners forked off either locally or on other machines to consume off the queue.
We use test-queue in a dozen different projects at GitHub and most CI runs finish in ~30s. The largest suite is for the github/github rails app which runs 30 minutes of tests in 90s.
Re: Running three hours of Ruby tests in under three minutes
#13I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?
This is a great question, and it's definitely a problem we have.
We don't have a single answer we use for every system we work on, but we employ a few common patterns, ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure. We have, for example, our own faked ISO-8583 [1] authorization service, which some of our tests run against to get a degree of end-to-end testing.
Back-testing is also incredibly valuable: We have repositories of every conversation or transaction we've ever exchanged with the banking networks, and when making changes to parsers or interpreters, we can compare their output against the old version on all of that historical data.
Re: Running three hours of Ruby tests in under three minutes
#14"This second round of forking provides a layer of isolation between tests: If a test makes changes to global state, running the test inside a throwaway process will clean everything up once that process exits."
But, then how do you catch bugs where shared mutable state is not compatible with multiple changes?
Re: Running three hours of Ruby tests in under three minutes
#15How much cheaper (in time, code, effort, complexity) would it be if:
- Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker.
- Their language runtime was AOT or JIT compiled, simply making everything faster to a degree that test execution could be reasonably performed on one (potentially large) machine.
- They used a language with a decent type system, significantly reducing the number of tests that had to be both written and run?
Re: Running three hours of Ruby tests in under three minutes
#16If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github. https://github.com/tmm1/test-queue One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.
test-queue supports minitest too, and follows the same basic design outlined in this article: a central queue sorted by slowest tests first, with test runners forked off either locally or on other machines to consume off the queue. We use test-queue in a dozen different projects at GitHub and most CI runs finish in ~30s. The largest suite is for the github/github rails app which runs 30 minutes of tests in 90s.
Thanks for the awesome work you have done on test-queue. We have really appreciated it at SchoolKeep.
Re: Running three hours of Ruby tests in under three minutes
#17How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…
Only if the early engineers could write in a language that they were as productive in as Ruby. Getting Stripe launched was the key thing Stripe needed to accomplish. Everything else follows from that.
Re: Running three hours of Ruby tests in under three minutes
#18Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?
Re: Running three hours of Ruby tests in under three minutes
#19How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…
Would it be cheap enough to encourage a re-write, re-engineer the server stack, and re-train employees? I doubt it.
I think it is an interesting question but a bad one most of the time unless you take into account all the other external factors that don't include the language it self.
Re: Running three hours of Ruby tests in under three minutes
#20How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…
Is it fair to assume that time, code, effort, and complexity would be some degree of cheaper?
May very well be more expensive. Language choice isn't a silver bullet.