Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How deep are your unit tests?

stackoverflow.com

Post reply on HN
Print view View original

21–30 of 69 posts

Re: How deep are your unit tests?

#21

Post by Firehed »

kibibyte wrote: post #19
Negitivefrags wrote: post #13

The thing I really dislike are unit testing styles that use mock objects a lot. I see some people say that you should test each object in isolation, mocking out any dependencies. That seems wrong headed to me. I prefer my unit tests to test everything all the way down. The only thing I would mock is the file system, which is useful for testing file loading code.

That's what I thought until my first summer internship at a Rails shop. Our unit test suite could take a very long time to run (on my Macbook Air dev machine, it would take 15-20 minutes to run). The slowdown came from testing ActiveRecord models with many dependencies, and that we were creating sample models. As a result, our tests hit the test database multiple times, which made them slow down. Our solution was to…

How many issues do you think may be hidden by mocking out the database layer? It's easy to write bad data due to a subtle bug in a query builder for example and then write garbage into the database. Mocking would make that significantly harder to catch.

Re: How deep are your unit tests?

#22

Post by Firehed »

MojoJolo wrote: post #2

I agree with the first answer. I'm not a fan of unit testing. I'm just doing it because it's a requirement (and we still have no QA). Why I'm not a fan? Because I'm doing my own tests. And probably it has a bias all over it. I know when it will have a successful run and I expect where it will fail. This is not the scenario that I wanted. (I know this is not a good testing scenario) In my opinion, a programmer must wr…

> In my opinion, a programmer must write a good code. And leave the testing to the QA.

Aside from the "not my problem" vibe that gives off, I strongly disagree. QA is helpful for covering weird user interactions and writing tests against bugs discovered in production, but you as the programmer should know what your code is supposed to do while you're writing it. Best write tests then rather than six months later when you've forgotten what it was for.

Re: How deep are your unit tests?

#23

Post by davidw »

I try and cover the easy 80% where it's helpful, and then specific things in the difficult 20% on an as-needed basis, or when bugs are discovered. It's an asymptotic curve: you want to spend the effort for the low-hanging fruit, but not get caught up chasing after that time-eating final bit of coverage.

Re: How deep are your unit tests?

#24

Post by microtonal »

tikhonj wrote: post #14
MojoJolo wrote: post #2

I agree with the first answer. I'm not a fan of unit testing. I'm just doing it because it's a requirement (and we still have no QA). Why I'm not a fan? Because I'm doing my own tests. And probably it has a bias all over it. I know when it will have a successful run and I expect where it will fail. This is not the scenario that I wanted. (I know this is not a good testing scenario) In my opinion, a programmer must wr…

This is one of the reasons I really like property-based testing à la QuickCheck[1]. The core idea is simple: you come up with an invariant for your code and the testing framework checks this invariant with randomly generated inputs. So when you're writing a test, you just have to come up with interesting invariants; you do not have to guess which inputs are edge cases. [1]: http://www.haskell.org/haskellwiki/Introduc…

I agree that QuickCheck is really nice. It helped me spot some errors in e.g. tree diff'ing code that I wrote. QuickCheck especially works well for fundamental data structures and algorithms, where invariants are clear and simple.

However, in other cases, the usage is less obvious. For instance, suppose that you are writing, say a URL parser. What do you want QuickCheck to generate and check? If you generate too broadly, the invariant check becomes a parser itself, if you generate too specifically you might introduce bias again.

Re: How deep are your unit tests?

#25

Post by jakejake »

Negitivefrags wrote: post #13

The thing I really dislike are unit testing styles that use mock objects a lot. I see some people say that you should test each object in isolation, mocking out any dependencies. That seems wrong headed to me. I prefer my unit tests to test everything all the way down. The only thing I would mock is the file system, which is useful for testing file loading code.

Technically it is called integration testing but I agree with you on the idea.

Unless there's a reason to mock something up (like simulating a fail condition from a DB or network call, or simulating other code that isn't finished yet) I prefer to just test everything as close as possible to what is actually running. The level of granularity, like the OP said, changes depending on the complexity and the critical nature of each part of the code.

Some people argue that if I make an error in function A and that causes a unit test fail for function B - that can mislead a developer into thinking function B has an error. I argue that a) it rarely misleads anybody for very long and b) if I had function B tested in total isolation with mocks then that error in function A might have slipped through unnoticed. It's tricky to mock up some combination that you weren't expecting to happen.

Obviously in ideal circumstances you would be doing both unit and integration testing. But, you have to draw a line somewhere otherwise you just write tests all day and get no production code running.

Re: How deep are your unit tests?

#26

Post by discreteevent »

Negitivefrags wrote: post #13

The thing I really dislike are unit testing styles that use mock objects a lot. I see some people say that you should test each object in isolation, mocking out any dependencies. That seems wrong headed to me. I prefer my unit tests to test everything all the way down. The only thing I would mock is the file system, which is useful for testing file loading code.

Here are qualities of good unit tests:

1) They run fast.

2) They help us localize problems.

In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly? Naturally, there is a continuum. Some tests are larger, and they use several classes together. In fact, they may seem to be little integration tests. By themselves, they might seem to run fast, but what happens when you run them all together? When you have a test that exercises a class along with several of its collaborators, it tends to grow. If you haven't taken the time to make a class separately instantiable in a test harness, how easy will it be when you add more code? It never gets easier.

- From "Working Effectively With Legacy Code" by Michael Feathers. (A really good book that actually does a better job of teaching design than many a book because like all good teaching it starts with the examples and abstracts from there, not the other way around.)

Re: How deep are your unit tests?

#27

Post by tikhonj »

microtonal wrote: post #24
tikhonj wrote: post #14

Earlier quoted context omitted.

This is one of the reasons I really like property-based testing à la QuickCheck[1]. The core idea is simple: you come up with an invariant for your code and the testing framework checks this invariant with randomly generated inputs. So when you're writing a test, you just have to come up with interesting invariants; you do not have to guess which inputs are edge cases. [1]: http://www.haskell.org/haskellwiki/Introduc…

I agree that QuickCheck is really nice. It helped me spot some errors in e.g. tree diff'ing code that I wrote. QuickCheck especially works well for fundamental data structures and algorithms, where invariants are clear and simple. However, in other cases, the usage is less obvious. For instance, suppose that you are writing, say a URL parser. What do you want QuickCheck to generate and check? If you generate too broa…

That's a good point. I remember having exactly that problem when writing tests for my toy language's parser. Generating random valid programs was actually tricky for several different reasons. In fact, I ultimately gave up on QuickCheck completely for that project, but mostly out of laziness.

Amusingly, I also wrote some tree diff code in Haskell recently. However, I did it rather poorly so I'm going to have to rewrite it soon (probably some time during winter break). I definitely agree that algorithms like this are particularly well-suited for QuickCheck.

Re: How deep are your unit tests?

#28

Post by FooBarWidget »

huhtenberg wrote: post #17

I find that simply assert'ing all non-trivial assumptions and invariants in the code is the form of unit testing that works the best in heck of a lot of cases. If there's need for explicit tests, just use the code in the right way and see it exit cleanly. Then use the code in the wrong way and see it assert (first replacing the assert() with a throw/longjmp to catch failures without aborting). Asserts also double as…

I've found asserts to be useful in addition to unit testing. The advantage of automated tests is that you don't have to run tests manually. Asserts merely tell you that something is wrong, they don't run test cases for you.

Re: How deep are your unit tests?

#29

Post by jiggy2011 »

> "If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it."

Hmm, that sounds like the old "You don't need to test , just be an awesome programmer duh!" argument.

The kind of mistakes you never anticipate making are probably the sort of stuff you should be testing for.

The amount of times I've had something fail on an edge case that turned out to be a typo in a var name on some edge case somewhere..

Re: How deep are your unit tests?

#30

Post by fruiapps »

Testing is something I do a lot, and like talking about a lot as well. There are no hard and fast answers to testing, like other stuff, there are lots of opinions(they are like assholes, everyone has them, and the others' stinks) about testing. So here goes mine(not necessarily correct, but worth putting them here). I am putting them as one liners, let me know your thoughts.

1. What do you mean by unit in unit tests?

An implementation of very basic feature, probably not more than 100 lines of code.

2. What do you test when you write a unit test?

We test the basic correctness, eg. given a url, if it exists assert 200, if not assert 404.

3. What should not be tested?

The inbuilt libraries should not be tested. Eg. If you are using a library say django, you should assume it comes pre tested and should not spend time writing tests for them.

4. What should be the depth of the tests?

There are no rules, but you should stop when you feel like you are humming inception theme.

If you want to read in slight more detail its http://www.blog.fruiapps.com/2012/09/An-intro-tutorial-to-te...

Post reply on HN