Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running three hours of Ruby tests in under three minutes

stripe.com

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Running three hours of Ruby tests in under three minutes

#91

Post by crdoconnor »

sanderjd wrote: post #30
crdoconnor wrote: post #7

Earlier quoted context omitted.

Could you not create mock embedded devices?

That's basically what we did, but more like mocking things at the network communication boundary. But for an integration test, it was often unsatisfactory, because of things behaving differently than they do in the real world. We also had a suite of automated tests that communicated with real devices in a lab, which were much better, but extremely hard to maintain. My general experience was: tenable, but tricky. So I…

Was it not possible to build upon the mock device to make it run more realistically?

I pretty much do this habitually now:

1. Get report of bug in production or on staging.

2. Write test to reproduce the bug.

3. 2/3 of the time get stuck because the environment isn't capable of mimicking the bug.

4. Build upon the integration testing environment to make it capable of mimicking the bug.

I find the counter intuitive part of integration testing is that step 4 ends up being where most of the work is required and far too many people just don't do it because they feel it's not a worthwhile investment.

I actually ended up writing an open source framework to handle a lot of the boilerplate (which no other frameworks do AFAIK). Especially making mock devices easier to write (see http://hitchtest.com/ and check out HitchSMTP).

Re: Running three hours of Ruby tests in under three minutes

#92

Post by kevan »

nelhage wrote: post #13
clayallsopp wrote: post #3

I'm super curious how Stripe approaches end-to-end testing (like Selenium/browser testing, but maybe something more bespoke too) My understanding is that they have a large external dependency (my term: "the money system"), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

This is a great question, and it's definitely a problem we have. We don't have a single answer we use for every system we work on, but we employ a few common patterns, ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure. We have, for example, our own faked ISO-8583 [1] authorization service, which some of our te…

> ranging from just keeping hard-coded strings containing the expected output, up to and including implementing our own fake versions of external infrastructure.

This sounds very familiar, we rely on external credit systems pretty heavily. We started by mocking service responses and including the response XML in our unit tests. Now we have a service simulator that returns expected values and has record/playback capability. It's not ideal and responses get outdated occasionally but we haven't found a more elegant way to handle it yet.

Re: Running three hours of Ruby tests in under three minutes

#93

Post by falsedan »

badmadrad wrote: post #47
falsedan wrote: post #35

Oh hey, we have the same sort of system here. It's 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI & scheduler & artifact distribution system (which we're probably going to replace with S3). We run selenium & unit tests as well as the integration tests. We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dock…

Have you considered another containerization solution like LXD. I feel like testing like this fits the "container hyper-visor" use case and this is what LXD is designed to do.

We tried docker, then had to drop back to running the tests outside of a container (some old technical decisions in the project under test made it hard to run in a container). It's been improved since then, and we're close to running in containers again.

Each executor gets a non-shared prod-like environment thanks to a handful of docker containers. The same setup is used for dev, so switching the testing environment to LXC would mean switching devs as well.

Re: Running three hours of Ruby tests in under three minutes

#94

Post by falsedan »

akoumjian wrote: post #90
falsedan wrote: post #35

Oh hey, we have the same sort of system here. It's 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI & scheduler & artifact distribution system (which we're probably going to replace with S3). We run selenium & unit tests as well as the integration tests. We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dock…

Anything significantly different in your Python implementation?

Hard to say anything more than what I posted without more details from Stripe.

Re: Running three hours of Ruby tests in under three minutes

#95

Post by bbuchalter »

ryanong wrote: post #10

If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github. https://github.com/tmm1/test-queue One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.

Could you share a bit more about this nginx proxy setup for static assets? Basically mimicking production env?

Re: Running three hours of Ruby tests in under three minutes

#96

Post by com2kid »

speedkills wrote: post #85
com2kid wrote: post #60

I am tired of this technology having to be re-invented time and time again. The best I ever saw was an internal tool at Microsoft. It could run tests on devices (Windows Mobile phones, but it really didn't care), had a nice reservation and pool system and a nice USB-->Ethernet-->USB system that let you route any device to any of the test benches. This was great because it was a heterogeneous pool of devices, with dif…

I agree. We already have projects like http://test-load-balancer.github.io but I have a feeling I will see five more posts on he in the next year about re-inventing this wheel and yet not see a single contribution to existing solutions like tlb. It must be a little depressing to build a really useful product you know many people need, give it away only hoping people will use it and be happy, then find out everyone wo…

> 3) I designed and implemented my own distributed test load balancing tool!

This, so many times over.

It doesn't help that when interviewing, I kinda-sorta want to know that the people I hire are capable of understanding systems from the ground up. The best way to demonstrate that is to go and build a system from the ground up...

TLB looks cool, nice to see that such a tool exists at least within one eco-system!

Re: Running three hours of Ruby tests in under three minutes

#97

Post by lawrencewu »

falsedan wrote: post #35

Oh hey, we have the same sort of system here. It's 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI & scheduler & artifact distribution system (which we're probably going to replace with S3). We run selenium & unit tests as well as the integration tests. We've noticed that starting and stopping a ton of docker containers in rapid succession really hoses dock…

hi frei

Re: Running three hours of Ruby tests in under three minutes

#98

Post by ersii »

nelhage wrote: post #49
heywire wrote: post #44

Earlier quoted context omitted.

>Back-testing is also incredibly valuable: We have repositories of every conversation or transaction we've ever exchanged with the banking networks, and when making changes to parsers or interpreters, we can compare their output against the old version on all of that historical data. Are you referring to test data or actual live transaction data? The latter would seem like a huge liability and target for hackers.

Live data, but they're stored redacted, and/or with sensitive data (e.g. credit card numbers) replaced with opaque tokens that reference an encrypted store that's carefully access-controlled.

Do you have any policy or decision made on how long you plan on storing that data? What I'm wondering, are the transactions currently "stored indefinitely"? (I'm referring to both data stores. The tokenized and the encrypted one)

Re: Running three hours of Ruby tests in under three minutes

#99

Post by beilabs »

ryanong wrote: post #10

If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github. https://github.com/tmm1/test-queue One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.

Really interested in this approach, can you point somewhere that talks further about this nginx proxy strategy?

Re: Running three hours of Ruby tests in under three minutes

#100

Post by bittersweet »

bbuchalter wrote: post #95
ryanong wrote: post #10

If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github. https://github.com/tmm1/test-queue One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.

Could you share a bit more about this nginx proxy setup for static assets? Basically mimicking production env?

Seconded, I have not thought about this before or come across this idea at all and certainly sounds interesting!
Post reply on HN