31–40 of 111 posts
This is an interesting and possibly overlooked problem with using slow languages like Ruby - your unit tests take forever to run. (unless you spend a lot of engineering effort on making them run faster, in which case they may run somewhat acceptably fast)
This is an interesting and possibly overlooked problem with using slow languages like Ruby - your unit tests take forever to run. (unless you spend a lot of engineering effort on making them run faster, in which case they may run somewhat acceptably fast)
This isn't just a problem with Ruby. Our ObjC test suite for a project I work on takes about 10 min to run, too.
Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies
Also tests like
def test1():
do_a()
check_condition_X()
def test2():
do_a()
check_condition_Y()
def test1():
do_a()
check_condition_X()
def test2():
do_a()
do_b()
check_condition_Y()
Then people wonder why it takes so much time?
Also helpful is if you can shutdown database setup for tests that don't need it
How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…
It would run the tests 10-100x faster per test, and also require less tests (due to having a real type system).
I do giggle a little when I see huge engineering hurdles people have to overcome because of the language that was chosen. Building an app that is going to scale to millions of users? May not want to use Ruby...
(Nothing against Stripe, I am a paying customer - love the product. I do suspect it would be easier to engineer on a better platform than RoR though).
We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dockerd, also that Jenkins' API is a lot slower than we expected for mostly-read-only operations.
Have you considered mesos?
"This second round of forking provides a layer of isolation between tests: If a test makes changes to global state, running the test inside a throwaway process will clean everything up once that process exits." But, then how do you catch bugs where shared mutable state is not compatible with multiple changes?
You write tests specifically for testing multiple changes. You shouldn't be testing changes to global state by seeing how multiple supposedly independent tests interact.
I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it's the "recommended TDD way") Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies Also tests like def test1(): do_a() check_condition_X() then def test2(): do_a() check_condition_Y() Or def test1(): do_a() check_condition_X() def test2(): do_a…
These are embarrassingly parallel problems, we just need better tools to fully saturate every core on every node in the test cluster.
I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it's the "recommended TDD way") Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies Also tests like def test1(): do_a() check_condition_X() then def test2(): do_a() check_condition_Y() Or def test1(): do_a() check_condition_X() def test2(): do_a…
Not only does the nesting help limit the amount of setup and teardown you do, but when broad-reaching functional changes hit you in version 2, 3, it's so much easier to reorganize your tests to get the pre- and post-conditions right when they are already grouped that way.
The sad thing is that it takes a few release cycles before you feel any difference at all, and a couple more before you're absolutely sure that there are qualitative differences between the conventions. So it seems like a pretty arbitrary selection process instead of an obvious choice.
Earlier quoted context omitted.
This isn't just a problem with Ruby. Our ObjC test suite for a project I work on takes about 10 min to run, too.
Our main product has approximately 5400 unit tests, over ~150 files. The test suite runs on a single computer in about 14s. This is one of the advantages of using a fast language (C++) with multithreading :)
On the other hand, using ruby I can have it continuously run the tests for the specific feature I'm working on without the long building step.
I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it's the "recommended TDD way") Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies Also tests like def test1(): do_a() check_condition_X() then def test2(): do_a() check_condition_Y() Or def test1(): do_a() check_condition_X() def test2(): do_a…
The time it saves me when I see 'test2' failed instead of 'test_enormous:137' failed is worth more than the marginal computation required. These are embarrassingly parallel problems, we just need better tools to fully saturate every core on every node in the test cluster.