Questions to ask yourself when writing tests • Hacker News Board

Re: Questions to ask yourself when writing tests

#21

Post by cle » Tue, Jan 02, 2018, 3:17 AM UTC

discreteevent wrote: post #8

ajmurmann wrote: post #5

Earlier quoted context omitted.

I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a…

Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming).

In this same vein, I propose another question:

   How similar does the test look to the production code?

These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.

Re: Questions to ask yourself when writing tests

#22

Post by spion » Tue, Jan 02, 2018, 3:18 AM UTC

atoko wrote: post #18

spion wrote: post #14

Here are 3 more questions: * Are these tests relying on API that is likely to change? * Can I make the API surface area used by all tests even smaller? * Can I make a library that wraps the existing API of the unit to get a smaller and/or more stable API for use in the tests only? These 3 help get tests that withstand refactors. An example: An acceptance test for an editing form is relying on the save button having a…

A small library function like document.getElementById?

The example wasn't perfect, but no, it wouldn't be that. What if in the future we have a screen that needs to show more than one save form? Then we'll need to stop using the "save" element id as a mechanism to denote save buttons; to do that, we need to update all the tests that rely on this mechanism.

The small library function would be `getSaveButton(form)` or even `save(form)` - now every form save test no longer encodes the knowledge of how a form's save buttons are made, whether that's by using a certain ID, class, text or anything else.

Now when we get that requirement for a screen with two forms, we'll no longer get mad and try to attack the idea (two save forms on one screen? that's inconsistent with our product, its confusing the users, etc etc) when the real reason is that it creates pain updating our tests. Instead we just update the save function.

The general idea is to encode the meaning of the test and separate the implementation details (clicking the save button might even be too concrete - "saving the form" is probably about the right level of abstraction). A good way to do this is to describe this test in prose and check if its encoding details that may vary - does this sound like something universally true / something that will be true forever, or accidentally true due to current circumstances?

Re: Questions to ask yourself when writing tests

#23

Post by simonhamp » Tue, Jan 02, 2018, 3:20 AM UTC

qaq wrote: post #11

simonhamp wrote: post #6

The most important question to ask when writing tests: Can I do this faster if I don’t write a test? The answer is always no. Even if you are the only person building something, future you will lick your boots clean in gratitude if there are tests. Because even the best developers have to work with their own code sometimes.

I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.

Of course, just because you haven’t personally written tests for parts of a stack doesn’t mean that you shouldn’t write tests for other parts, especially the parts you’ve created.

Also don’t forget that certain things like Linux are extremely battle-tested. It’s generally unlikely that any piece of software you write will receive that much real-world testing, so a growing set of automated tests will help you cover your own arse.

Re: Questions to ask yourself when writing tests

#24

Post by yawaramin » Tue, Jan 02, 2018, 3:20 AM UTC

Looks like Michal has been immersed in Haskell for at least the past year. I wonder if he will have something to say about balancing testing and coverage with static typing.

Re: Questions to ask yourself when writing tests

#25

Post by lambdabit » Tue, Jan 02, 2018, 3:20 AM UTC

Here's one more. Is maintaining this test more time and work than testing manually?

Re: Questions to ask yourself when writing tests

#26

Post by securingsincity » Tue, Jan 02, 2018, 3:25 AM UTC

I'll add a couple more

Can I run these tests more than once? Will they ever go stale? Can I run these tests and they'll clean themselves up?

Having to update tests because they didn't take into account the date changing (Happy birthday Joe Test!) or manually cleaning up data is a huge time suck.

Re: Questions to ask yourself when writing tests

#27

Post by u801e » Tue, Jan 02, 2018, 3:27 AM UTC

cle wrote: post #21

discreteevent wrote: post #8

Earlier quoted context omitted.

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a…

Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to re…

> I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details

I think this comes from "unit tests" where they have assertions such as

  assertThisOtherSpecificMethodNameWasCalledWithTheseSpecificallyNamedParameters

If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. This would also eliminate the need for mocking since the methods that are tested have no side effects.

For testing dependencies, an environment where a test instance of the database, API, or other external service is needed. Then the code could be tested against an actual implementation rather than a mock of it.

Re: Questions to ask yourself when writing tests

#28

Post by darioush » Tue, Jan 02, 2018, 4:20 AM UTC

cle wrote: post #21

discreteevent wrote: post #8

Earlier quoted context omitted.

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a…

Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to re…

Another way they prevent refactoring is by creating the work for you to implement the same type / number of tests on your new code so your code review doesn't look like it's deleting 100 tests.

Re: Questions to ask yourself when writing tests

#29

Post by querulous » Tue, Jan 02, 2018, 4:21 AM UTC

discreteevent wrote: post #8

ajmurmann wrote: post #5

Earlier quoted context omitted.

I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a…

what's wrong with just deleting these now-irrelevant tests?

Re: Questions to ask yourself when writing tests

#30

Post by ajmurmann » Tue, Jan 02, 2018, 4:28 AM UTC

discreteevent wrote: post #8

ajmurmann wrote: post #5

Earlier quoted context omitted.

I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a…

I've seen this with out differently on different code bases. Particularly on Rails code bases I've seen some of the symptoms you describe. On more recent Java projects I've seen the opposite where code that needed to change actually just ended up getting swapped out. The "only one reason to change"/open-closed dream come true. Old classes and unit tests just got thrown out. On those acceptance tests were the pain point when behavior was changed on purpose.

I'm still not sure how to get the good outcome every time. Maybe it's just well designed interfaces ‍️