Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running three hours of Ruby tests in under three minutes

stripe.com

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Running three hours of Ruby tests in under three minutes

#61

Post by birdsbolt »

schneems wrote: post #56
yjgyhj wrote: post #55

One thing I've noticed since coding with immutable data structures & functions (rather than mutable OOP programs) is how tests run really fast, and are easy to run in parallell. I/O only happens in a few functions, and most other code just takes data in -> transforms -> returns data out. This means I only have few functions that need to 'wait' on something outside of itself to finish, and much lesser delays in the co…

Immutable data structures give you easy parallelism, however there's a hidden runtime cost: you have to allocate way more objects. For example, I was able to save a ton of object allocations here: https://github.com/mime-types/ruby-mime-types/pull/93 mostly by mutating. For tasks that are not easily parallelizable it may be slower to use immutable structures. I mostly only ever hear about how fast FP languages are, s…

Allocations don't have to be expensive if your GC is smart. Smart as C++ destructors positioning :D

Re: Running three hours of Ruby tests in under three minutes

#62

Post by yjgyhj »

schneems wrote: post #56
yjgyhj wrote: post #55

One thing I've noticed since coding with immutable data structures & functions (rather than mutable OOP programs) is how tests run really fast, and are easy to run in parallell. I/O only happens in a few functions, and most other code just takes data in -> transforms -> returns data out. This means I only have few functions that need to 'wait' on something outside of itself to finish, and much lesser delays in the co…

Immutable data structures give you easy parallelism, however there's a hidden runtime cost: you have to allocate way more objects. For example, I was able to save a ton of object allocations here: https://github.com/mime-types/ruby-mime-types/pull/93 mostly by mutating. For tasks that are not easily parallelizable it may be slower to use immutable structures. I mostly only ever hear about how fast FP languages are, s…

Yes, mutating something at one place in memory is more efficient, because you don't need to allocate new memories.

I don't know all to much about other functional languages, as I learned perl -> ruby -> javascript -> little bit C & Java & Go -> now doing Clojure. But I find Clojures collection data structures interesting. The vector (collection like lists or array) type look immutable, but under the hood are trees. When you append to a vector, you seem get a new vector returned.

In reality, you added the new element as a node in a tree. Then just modified pointers to that the new and old version share almost all of the pointers & allocations. With simple arrays or lists, you would allocate every element anew.

Idk if I can properly explain. Found this blog post very interesting and easier to follow: http://hypirion.com/musings/understanding-persistent-vector-...

Personally, most things in business I find easily parallelisable. You mostly decouple the I/O parts of something with the logic parts of it. But yeah I still have much to learn. Thanks for the link! Interesting :)

Re: Running three hours of Ruby tests in under three minutes

#63

Post by hueving »

com2kid wrote: post #60

I am tired of this technology having to be re-invented time and time again. The best I ever saw was an internal tool at Microsoft. It could run tests on devices (Windows Mobile phones, but it really didn't care), had a nice reservation and pool system and a nice USB-->Ethernet-->USB system that let you route any device to any of the test benches. This was great because it was a heterogeneous pool of devices, with dif…

>I am tired of this technology having to be re-invented time and time again.

So did Microsoft open source this? If not, quit complaining. Just because you saw a massive software engineering company doing something better doesn't mean everyone else who doesn't have access to it sucks for not reaching parity.

Re: Running three hours of Ruby tests in under three minutes

#64

Post by teacup50 »

pekk wrote: post #21
teacup50 wrote: post #15

How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…

Thread-based concurrency based on shared mutable state doesn't reduce complexity.

Thread-based concurrency doesn't require shared mutable state at the application implementation level.

Re: Running three hours of Ruby tests in under three minutes

#65

Post by teacup50 »

100k wrote: post #17
teacup50 wrote: post #15

How much cheaper (in time, code, effort, complexity) would it be if: - Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity and actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker. - Their language runtime was AOT or JIT compiled, s…

Only if the early engineers could write in a language that they were as productive in as Ruby. Getting Stripe launched was the key thing Stripe needed to accomplish. Everything else follows from that.

There are certainly enough languages to choose from.

Re: Running three hours of Ruby tests in under three minutes

#66

Post by hinkley »

junto wrote: post #42
hinkley wrote: post #41

needle scratching on record They have an average of 9 assertions per test case. I think I may see part of their problem.

I'm not sure if you are talking from a performance perspective or a conceptual perspective, but this provides a useful discussion on multiple assertions: http://programmers.stackexchange.com/questions/7823/is-it-ok... My 2 cents is that multiple assertions are legitimate, as long as they prove a singluar assumption. Hence (as per the test on that page), this is a valid use of multiple assertions: [Test] public void V…

That's two asserts, and yes you are essentially testing the same concept, which I'm comfortable with as long as it's not a regular thing. People go through all sorts of gymnastics to convince themselves "it's one thing" and I find it exhausting, especially since fixing the problem is usually easier than the rationalizing.

If multiple asserts is a regular thing, you can either start breaking down your tests, or write a custom matcher. The custom matcher gives you better diagnostics when it breaks, so is probably the way to go.

Assert.That(value, Is.Within(10, 100)); // matcher generates error message

Re: Running three hours of Ruby tests in under three minutes

#67

Post by twerquie »

hinkley wrote: post #38
raverbashing wrote: post #33

I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it's the "recommended TDD way") Things like using the same setup function for every test and setting up/tearing down for every test regardless of dependencies Also tests like def test1(): do_a() check_condition_X() then def test2(): do_a() check_condition_Y() Or def test1(): do_a() check_condition_X() def test2(): do_a…

mocha and jasmine (in the node/javascript space) support nested setup and teardown methods and it's been really challenging for me to go back to using other frameworks, languages. Not only does the nesting help limit the amount of setup and teardown you do, but when broad-reaching functional changes hit you in version 2, 3, it's so much easier to reorganize your tests to get the pre- and post-conditions right when th…

I'm not sure if you're talking about the pain of going back to testing ruby / rails after using mocha / node, but I feel that specific pain, especially on projects with old-school Rails purists who insist on Test::Unit style. Switching to rspec gives you nested describe blocks with shared setup and teardown steps, as nice as mocha. Minitest has this BDD style built in too, but somehow the way Rails ties it in makes it difficult or impossible to take advantage of.

Re: Running three hours of Ruby tests in under three minutes

#68

Post by hinkley »

junto wrote: post #42
hinkley wrote: post #41

needle scratching on record They have an average of 9 assertions per test case. I think I may see part of their problem.

I'm not sure if you are talking from a performance perspective or a conceptual perspective, but this provides a useful discussion on multiple assertions: http://programmers.stackexchange.com/questions/7823/is-it-ok... My 2 cents is that multiple assertions are legitimate, as long as they prove a singluar assumption. Hence (as per the test on that page), this is a valid use of multiple assertions: [Test] public void V…

[Edit] Thanks for the link. I have a whole bunch of comments in there to upvote. Guess my evening is planned :) [Edit]

I would also like to point out that ranges, like the one in your example, are almost always a symptom of an unstable test to begin with. I'd want to know why you're providing a range. Does the test blow up if another server is running tests at the same time? Let's fix that so the tests actually fail when there is a failure.

Now, there are lots of matchers that misbehave for corner case inputs, and an assert like "Make sure there's text, that it's a number, and that the number equals 10" may be necessary in order to prove that "10" appears, especially when you invert the test an say "Make sure the number isn't 10". And in this case I would say "write us a better matcher so that everyone can benefit from you figuring out how to do this".

This should go without saying, but I feel I have to repeat it every time there's an audience:

Green is not the end goal of testing. Red when there is an actual problem is the end goal of testing. Anything else is a very expensive way to consume resources.

Post reply on HN