needle scratching on record
They have an average of 9 assertions per test case. I think I may see part of their problem.
Running three hours of Ruby tests in under three minutes
41–50 of 111 posts
Re: Running three hours of Ruby tests in under three minutes
#42needle scratching on record They have an average of 9 assertions per test case. I think I may see part of their problem.
I'm not sure if you are talking from a performance perspective or a conceptual perspective, but this provides a useful discussion on multiple assertions:
http://programmers.stackexchange.com/questions/7823/is-it-ok...
My 2 cents is that multiple assertions are legitimate, as long as they prove a singluar assumption. Hence (as per the test on that page), this is a valid use of multiple assertions:
[Test]
public void ValueIsInRange()
{
int value = GetValueToTest();
Assert.That(value, Is.GreaterThan(10), "value is too small");
Assert.That(value, Is.LessThan(100), "value is too large");
}Re: Running three hours of Ruby tests in under three minutes
#43Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn't that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?
how do you trust the hosted CI company not to look at your code? Contracts, not firewalls, make the world go round.
To be fair a contract does not guarantee the security framework of the company you are contracting, which means your code is only as safe as their weakest link.
Re: Running three hours of Ruby tests in under three minutes
#44I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?
This is a great question, and it's definitely a problem we have. We don't have a single answer we use for every system we work on, but we employ a few common patterns, ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure. We have, for example, our own faked ISO-8583 [1] authorization service, which some of our te…
>Back-testing is also incredibly valuable: We have repositories of every conversation or transaction we've ever exchanged with the banking networks, and when making changes to parsers or interpreters, we can compare their output against the old version on all of that historical data.
Are you referring to test data or actual live transaction data? The latter would seem like a huge liability and target for hackers.
Re: Running three hours of Ruby tests in under three minutes
#45I wrote a rubygem called cloudspeq (http://github.com/meesterdude/cloudspeq) that distributes rails rspec spec's across a bunch of digital ocean machines to reduce test execution time for slow test suits in dev.
one of the things I did that may be of interest is to break up spec files themselves to help reduce hotspots (or dedicate a machine to it specifically)
Not as complex or as robust as what they did, but it works!
Re: Running three hours of Ruby tests in under three minutes
#46Earlier quoted context omitted.
Our main product has approximately 5400 unit tests, over ~150 files. The test suite runs on a single computer in about 14s. This is one of the advantages of using a fast language (C++) with multithreading :)
Not exactly a fair comparison, because you have to compile the changed files first and link which I'm sure takes longer than 14s. On the other hand, using ruby I can have it continuously run the tests for the specific feature I'm working on without the long building step.
Compiling a changed file and linking takes about 5s.
Re: Running three hours of Ruby tests in under three minutes
#47Oh hey, we have the same sort of system here. It's 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI & scheduler & artifact distribution system (which we're probably going to replace with S3). We run selenium & unit tests as well as the integration tests. We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dock…
Have you considered another containerization solution like LXD. I feel like testing like this fits the "container hyper-visor" use case and this is what LXD is designed to do.
Re: Running three hours of Ruby tests in under three minutes
#48I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it's the "recommended TDD way") Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies Also tests like def test1(): do_a() check_condition_X() then def test2(): do_a() check_condition_Y() Or def test1(): do_a() check_condition_X() def test2(): do_a…
The time it saves me when I see 'test2' failed instead of 'test_enormous:137' failed is worth more than the marginal computation required. These are embarrassingly parallel problems, we just need better tools to fully saturate every core on every node in the test cluster.
Though I certainly don't advocate a test that goes to 137 lines, I think the point of having to guess what the test is doing only by the name/messages is moot, you'll end up checking the test source code to see what it is doing exactly
Re: Running three hours of Ruby tests in under three minutes
#49Earlier quoted context omitted.
This is a great question, and it's definitely a problem we have. We don't have a single answer we use for every system we work on, but we employ a few common patterns, ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure. We have, for example, our own faked ISO-8583 [1] authorization service, which some of our te…
>Back-testing is also incredibly valuable: We have repositories of every conversation or transaction we've ever exchanged with the banking networks, and when making changes to parsers or interpreters, we can compare their output against the old version on all of that historical data. Are you referring to test data or actual live transaction data? The latter would seem like a huge liability and target for hackers.
Live data, but they're stored redacted, and/or with sensitive data (e.g. credit card numbers) replaced with opaque tokens that reference an encrypted store that's carefully access-controlled.
Re: Running three hours of Ruby tests in under three minutes
#50Earlier quoted context omitted.
This is a great question, and it's definitely a problem we have. We don't have a single answer we use for every system we work on, but we employ a few common patterns, ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure. We have, for example, our own faked ISO-8583 [1] authorization service, which some of our te…
>Back-testing is also incredibly valuable: We have repositories of every conversation or transaction we've ever exchanged with the banking networks, and when making changes to parsers or interpreters, we can compare their output against the old version on all of that historical data. Are you referring to test data or actual live transaction data? The latter would seem like a huge liability and target for hackers.
[deleted]