Write tests. Not too many. Mostly integration • Hacker News Board

Re: Write tests. Not too many. Mostly integration

#321

Post by throwaway5752 » Sat, Oct 28, 2017, 12:26 AM UTC

sheepmullet wrote: post #227

throwaway5752 wrote: post #11

Why does everyone rethink a working strategy. Write lots of unit tests that are fast. Write a good amount of integration tests that are relatively fast. Write fewer system integration tests that are slower. The testing pyramid works. He even talks about it in this post, and then ignores the point of it. You write lots of unit tests because you can run them inline pre-commit or in a component build. If you integration…

> The testing pyramid works. The testing pyramid is built around a lot of assumptions which are often not true. For example I run our ~10,000 integration tests in under two minutes on our large enterprise codebase. In recent years it has become possible to have fast integration tests. I've worked on other apps that take 5+ minutes just to start up and integration tests can take hours. Applying the same testing strate…

The pyramid isn't a law, it's just a heuristic that says to have more low level tests that are faster than high level tests that are slower. The unit/integration/system integration division tends to be correct, but isn't always. It just reminds us that there is time/compute scarcity, and to maximize those resources for optimal roi. And yes, a mobile app != a PaaS platform != system software and adapt the principles sensibly to the situation.

Seriously, though, I salute you on those integration test numbers. I assume containers are involved?

Re: Write tests. Not too many. Mostly integration

#322

Post by jcrben » Sat, Oct 28, 2017, 12:26 AM UTC

titzer wrote: post #250

jcrben wrote: post #224

Earlier quoted context omitted.

Are you suggesting that the Linux kernel is unit tested? Last I checked, I couldn't find them. And discussions online say the same (e.g., https://news.ycombinator.com/item?id=9543336 and https://news.ycombinator.com/item?id=9544306 ). Kernel bugs tend to show up in userspace. Fortunately, integration-oriented projects have arisen more recently such as https://kernelci.org/ and https://github.com/os-autoinst/openQA/ A…

https://cs.chromium.org/chromium/src/v8/test/unittests/

The ones I glanced at appear to test output without much mocking, which isn't as tightly coupled as unit tests commonly end up.

Re: Write tests. Not too many. Mostly integration

#323

Post by emodendroket » Sat, Oct 28, 2017, 1:16 AM UTC

seanmcdirmid wrote: post #316

emodendroket wrote: post #240

Earlier quoted context omitted.

I think I am not mistaken in saying extension methods, like lambda functions, were invented primarily for the use case of Linq. Even if they weren't, that's how Linq is implemented, so extension methods serve more than that "one purpose" if you don't insist on writing C# in the style of C# 2.0.

They came out at the same time, I’m sure there was some influence between them (Mads Tergesen would know better). However, all the functionality added in could have been done with static methods, just with more verbose syntax. LINQ query syntax could have been special cases. Anyways, I like what they came up with, it’s very versatile.

It's not clear that would have been much less work

Re: Write tests. Not too many. Mostly integration

#324

Post by shados » Sat, Oct 28, 2017, 3:51 AM UTC

hinkley wrote: post #302

shados wrote: post #249

I go the complete opposite way. I've tried various testing strategies over 15~ different companies in all sorts of environments, and unit tests are the only thing that really work (IF you can convince the team to do it...and that's a big IF). The article starts with a point I agree with: the lower in the pyramid, the cheaper the tests but the lower the confidence level they bring. That's true. Where I disagree is how…

The second group I worked with that was earnestly interested in mature testing developed the 5/8ths rule. To move a test one level down the pyramid, it takes about 5x as many tests. But the tests run 8 times as fast. So moving a test down takes more than 35% off the run time, and it fails the build minutes sooner . If you drop it down two levels it's 60% off the run time. Interesting enough on its own, but maintainin…

Yup. I think one big issue is that a E2E or an integration test is useful on its own, while a single unit test is almost totally worthless. You don't have confidence of anything until at least 50% (and at 80% you have almost perfect confidence).

So when people get started, especially on an old code base, they feel its pointless and doesn't pay off. Can't blame them, I suppose.

Good that you bring up build time. I forgot to mention that. We have repos with thousands of tests where the whole suite runs in <1 minute and gives us very high confidence (actually the only other tests we run on that repo are visual regression tests for CSS, and even E2E tests don't catch those issues...). During that time I'm watching other teams waiting 20 minutes on their integration test suite. Nope nope nope.

Re: Write tests. Not too many. Mostly integration

#325

Post by Vinnl » Sat, Oct 28, 2017, 8:26 AM UTC

breatheoften wrote: post #314

Vinnl wrote: post #85

Earlier quoted context omitted.

> to get the most value, test suites should accelerate the time to useful feedback Well, they should also optimise the usefulness of the feedback they provide. Typically, tests higher up the pyramid are also more brittle (e.g. end-to-end tests might fire up an entire browser and Selenium), and thus are more likely to fail when in actuality, nothing is wrong. That's an additional reason for limiting the number of thos…

Brittle tests seem not useful in general though aren't they? I'm not sure its necessarily true that brittleness must correlate with height in pyramid or execution time -- in my experience brittleness correlates with selenium more than it does pyramid height (that's a statement about selenium more than it is a statement about any particular category of testing pyramid). Its possible to write very useful non-brittle te…

No they're not.

But yes, Selenium is brittle. That said, Google engineers actually did some investigation into this, and although I think their methods were probably a bit heavyweight, they did conclude that it's mostly RAM use that leads to brittleness.

[1] https://testing.googleblog.com/2017/04/where-do-our-flaky-te...

Re: Write tests. Not too many. Mostly integration

#326

Post by profalseidol » Sat, Oct 28, 2017, 9:03 AM UTC

e12e wrote: post #266

profalseidol wrote: post #202

Earlier quoted context omitted.

I agree with this. Although, if you offshore development here in the third-world where internet gets slower everyday. Running an integration test that queries to Amazon RDB can take forever. I hope this issue gets a spotlight and be noted that integration tests in third world countries is very very slow. And this high cost should be included in the estimates. To give you an idea, here it takes AT LEAST 5 seconds to l…

> Running an integration test that queries to Amazon RDB can take forever. Why wouldn't you run your test on aws if it needs to integrate with rdb anyway? Or are your code changes so massive that "git push" is slow?

No massive.. But yes, git push takes a few seconds, but bearable. Once pushed, I'd need to do some SSH into the Jenkins server, so you run only your brand new integration test. Running the test there is super fast, but everything else including typing one character in PuTTY is slowed dowwn.

All this while you are expected to fix 10 tickets for the whole day plus anything that goes wrong in production.

Re: Write tests. Not too many. Mostly integration

#327

Post by cjsuk » Sat, Oct 28, 2017, 10:03 AM UTC

quickthrower2 wrote: post #311

cjsuk wrote: post #133

Earlier quoted context omitted.

It's not that simple and it's not a fact. I'm an advanced user of functional languages as well and have written an entire scheme implementation before. I only semi-agree. That's slightly disingenuous representation of functional languages which have more than a few pitfalls. They certainly aren't the silver bullet and they really do not scale to the same height and complexity of the problem domain as the OO languages…

To be clear I'm not saying functional is always better. I'm saying there are other possibilities and dogma is bad. The Haskell community has its dogmas too. And it's fair share of "let's do this simpler" blog posts. As an aside Haskell has many equivalents of dependency injection and sugars to help and you can "inject all the things" there too. My point is to think of the problem you are trying to solve, rather than…

Breaking the rules is good when appropriate. Problem is those rules are pretty amazingly good. I went through a weird phase of denial and ended up back where I started before I applied the aforementioned rules.

Every exception I have shot myself.

Re: Write tests. Not too many. Mostly integration

#328

Post by cjsuk » Sat, Oct 28, 2017, 10:03 AM UTC

gurkendoktor wrote: post #308

cjsuk wrote: post #80

Earlier quoted context omitted.

Exactly this. Confidence is everything. I can actually write entire features with appropriate test coverage from the ground up and they work first time and have close to zero defects in production. It's amazing when you spend 5-6 days writing code that does nothing and at the last moment, everything slots together with a few integration tests and wham, feature done. Not talking trivial stuff here either; big integrat…

> You see a lot of people arguing against this but I'm going to be honest, they churn out a lot of stuff that doesn't actually work. My anecdata cancels out your anecdata. The TDD practitioners that I've met have, without exception, written code that worked fine for only the one case that they've tested. Example: They'd test a method for sending a message with the string "hello". Turns out the method didn't URL-encod…

That will be because you worked with dumbasses.

If you only test the expected outcome you are a dumbass.

Re: Write tests. Not too many. Mostly integration

#329

Post by piaste » Sat, Oct 28, 2017, 1:22 PM UTC

crdoconnor wrote: post #115

piaste wrote: post #74

Earlier quoted context omitted.

> If your app relies heavily on using a database, your app naturally integrates with a database then it makes no sense to test without it. You are intentionally avoiding testing in a way that will pick up bugs. Also, with Docker it's now actually feasible to automatically test against a real database at a reasonable speed. A Postgres container spins up in a couple of seconds, a SQL Server one in a little over four.

That has nothing to do with docker, really. I run postgres standalone on my laptop and it starts in < 1 second.

Sure, but I presume that doesn't include installation time.

Re: Write tests. Not too many. Mostly integration

#330

Post by piaste » Sat, Oct 28, 2017, 1:33 PM UTC

GordonS wrote: post #242

crdoconnor wrote: post #115

Earlier quoted context omitted.

That has nothing to do with docker, really. I run postgres standalone on my laptop and it starts in < 1 second.

I guess they meant so your tests can start with a blank or reproducible state. But you can of course achieve the same by running a script before your tests start. There are also some frameworks for doing this sort of thing too, such as Fixie for .NET

I've done "write a script to reset the database" before, although not for Pgsql. The effort and potential snags involved make it nowhere near as trivial as docker rm && docker run.

There are also other scenarios that become really simple with disposable DB instances. Want to test a remote data transfer feature? Just spin up two databases.