Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why TDD isn't crap

hillelwayne.com

Post reply on HN
Print view View original

161–170 of 171 posts

Re: Why TDD isn't crap

#161

Post by Vinnl »

the_af wrote: post #137
Vinnl wrote: post #87

Earlier quoted context omitted.

Note that mutation testing is mostly just an advanced (and more brittle) form of code coverage. The flip side is that code coverage reports can also be considered tests of your tests. Not exhaustive, but I'd argue that unit tests hardly ever are as well - and that's OK.

I disagree. Coverage tests how much of your code is exercised by your tests. Mutation testing is (one way) of testing how effective your tests are -- i.e. how sensitive they are to bugs in your code. It follows that good tests must have both good coverage and good effectiveness, but the two are not the same.

The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it.

Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code.

It might sound pedantic, but this way of thinking shows you that it's not turtles all the way down: if they'd really be "tests of the tests", then you might also want "tests of the tests of the tests". And since we obviously don't want to keep doing that, you might conclude that we don't need "tests of our tests" as well.

But since they're simply measurements of test quality, we can see that both test coverage and mutation testing can be useful and not a never-ending story.

Re: Why TDD isn't crap

#162

Post by bunderbunder »

humanrebar wrote: post #160
echlebek wrote: post #157

Earlier quoted context omitted.

Static languages make refactoring easier, not harder. The more information you have at compile time, the more automated tooling can do for you.

The more compile time information you have, the greater coupling you have as well. Make User subclass Entity for polymorphism reasons and now taking a User parameter makes your business logic depend explicitly on your ORM.

But now the conversation's going in a circle. In a decent static language, you should never have to run into that particular problem.

Assuming it's a modern static OO language, your business logic should depend on a User interface, so that it never has to take a dependency on implementation details like that. Even if User was a class beforehand, you can easily extract an interface at a later date, when you find that you need to avoid some tight coupling.

Don't blame the gun for what happens when you point it toward your foot and pull the trigger.

Re: Why TDD isn't crap

#163

Post by lowbloodsugar »

Verdex_3 wrote: post #146
lowbloodsugar wrote: post #128

Earlier quoted context omitted.

If someone gave me that test, I'd implement the code as "return input;" Forget that you, a human, thinks you know what an ISorter should do. What do the tests demand that you do? Doing TDD, the goal, when coding, is to write the minimal amount of broken code that makes the test pass. If you passed me {4,3,2,1}, then my code would be "return {1,2,3,4};". If you wrote a second test that passed me {6,5,4,3}, my code wou…

And that works fine ... as long as you know all of the cases that will cause the minimal amount of broken code to fail in a test case. What happens if you're implementing something like a unification algorithm, but you don't know about the occurs check. You'll progressively add test cases that break your code and eventually stop. Until an end user creates a query that contains itself and your algorithm fails to termi…

I can only suggest that you commit to it for some considerable time, such as six months. I, too, thought it was, at first, a fucking stupid idea, and then, when people I respected continued to advocate it, grudgingly tried it, and thought it was a pain in the ass - this was in 2006. However, the number of times where I've written some code that I thought worked, and then commented it out and rewrote it using TDD, and found bugs, is too high. Its rare, but its too high. So either I'm not that great a programmer, or TDD is useful. Another way to think of TDD is rubber ducking, perhaps.

So you're in a local maximum, and to get to the next, higher, maximum, you're going to have to go through a trough.

Re: Why TDD isn't crap

#164

Post by the_af »

Vinnl wrote: post #161
the_af wrote: post #137

Earlier quoted context omitted.

I disagree. Coverage tests how much of your code is exercised by your tests. Mutation testing is (one way) of testing how effective your tests are -- i.e. how sensitive they are to bugs in your code. It follows that good tests must have both good coverage and good effectiveness, but the two are not the same.

The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it. Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code. It might sound pedantic, but this way o…

I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other:

- Coverage measures how much of your code is exercised by the tests. It does NOT measure test effectiveness (it's not like mutation, only "less/more brittle"). This can be trivially shown by writing tests that exercise all of your code but have no asserts (or trivial asserts such as 1 == 1). Unsurprisingly, this kind of obviously ineffective test code is often find in the wild (mostly written by junior devs), but less obvious cases of ineffective tests are also common, such as failing to test border conditions. This happens and it's more common than we'd like.

- Mutation testing is a way (but not the only one!) to measure how effective your tests are at actually finding bugs. That this technique exists shows that there is indeed a need to "test the tests", i.e. measure test quality & effectiveness. Production code is NOT "the test of the tests", as someone up this thread erroneously said.

This is like measuring altitude and airspeed: yes, they both measure something useful about your airplane, but they measure different things!

I confess I did not understand the rest of your post.

Re: Why TDD isn't crap

#165

Post by nightski »

hacker_9 wrote: post #58
quicklime wrote: post #32

Earlier quoted context omitted.

I've always thought that the bulk of the value of tests is for the unfortunate person who has to refactor or extend your code years down the track. By that logic, if you write testable, but untested code, then you're still making it difficult for people to refactor later. This applies even if the code is well designed and uncoupled.

You don't even have to think that far ahead - consider some code you wrote in week 1, that was then modified by a colleague in week 2, now you return to the code in week 3 - what does the code do? Any mental model you had at week 1 is now irrelevant as the code has changed in unknown ways. What is the current intent of the code? Do you feel safe modifying it, despite the fact you no longer know how to manually test a…

What do I do? Review the changes in git. Am I the only one that doesn't have much faith in unit tests preventing a regression even on a well covered project?

Re: Why TDD isn't crap

#166

Post by ellius »

convolvatron wrote: post #159
barrkel wrote: post #127

Earlier quoted context omitted.

I think TDD (or rather, unit tests): 1) ... over-emphasise the importance of reduced coupling, and can actually increase the chances of integration failures. Why? Because components are tested in isolation rather than together, but are still considered "tested" - especially when using stubs and mocks. I've seldom seen mocking used well, it almost always over-specifies implementations. 2) ... increase the tested "surf…

I worked at a place recently where all of these factors, but especially (3) turned out to be a huge cost for an ostensibly simple service that it severely hampered any further development. Since it was a hard TDD shop, any discussion of cutting out all the intermediate layering was completely off the table. Internal structure was easily 75% of the size of the project. After all, a lot of effort had gone into building…

“Any technique, however worthy and desirable, becomes a disease, when the mind is obsessed with it.” —Bruce Lee

Re: Why TDD isn't crap

#167

Post by Vinnl »

the_af wrote: post #164
Vinnl wrote: post #161

Earlier quoted context omitted.

The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it. Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code. It might sound pedantic, but this way o…

I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other: - Coverage measures how much of your code is exercised by the tests. It does NOT measure test effectiveness (it's not like mutation, only "less/more brittle"). This can be trivially shown by writing tests that exercise all of your code but have no asserts (or trivial asserts such as 1 == 1). Unsurprisingly, t…

I didn't mean coverage measurement is a way of doing mutation testing, I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness.

Yes, it can be inaccurate, as your trivial example shows, but it does guide you towards unhandled test cases (i.e. where your tests are ineffective). Likewise, mutation testing guides you towards unhandled test cases (including ones that coverage can't detect).

Re: Why TDD isn't crap

#168

Post by the_af »

Vinnl wrote: post #167
the_af wrote: post #164

Earlier quoted context omitted.

I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other: - Coverage measures how much of your code is exercised by the tests. It does NOT measure test effectiveness (it's not like mutation, only "less/more brittle"). This can be trivially shown by writing tests that exercise all of your code but have no asserts (or trivial asserts such as 1 == 1). Unsurprisingly, t…

I didn't mean coverage measurement is a way of doing mutation testing, I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness. Yes, it can be inaccurate, as your trivial example shows, but it does guide you towards unhandled test cases (i.e. where your tests are ineffective). Likewise, mutation testing guides you towards unhandled test cases (including ones th…

> I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness.

The problem is that coverage alone is a very poor way of measuring effectiveness, which is why other techniques -- including, but not limited to, mutation testing -- are needed. This is nothing new: limitations of coverage are well known in software engineering.

Re: Why TDD isn't crap

#170

Post by ozim »

hacker_9 wrote: post #40
ozim wrote: post #30

Earlier quoted context omitted.

Pick the tool for the job, if your experience is with things that last 5+ years then ok. I work on a system which is 5 years old and all things from 5 years ago are not relevant. Actually I am working on it only for 3 years now but we pretty much each year rebuild whole thing. With minor things going in and out. Of course we spend a lot of money on automated tests but those tests had to be thrown away because of amou…

So you disagree with TDD, but you need to rebuild your system from scratch every 5 years? I feel like this is an argument for TDD. As for throwing away tests; Unit tests are meant to be pretty simple - rule of thumb is you can run a thousand tests in ten seconds. Arrange, Act, Assert - they don't need to be complex, they just need to imprint the intent into the codebase. If the intent changes, by all means remove the…

We rebuild because business needs change, because compliance and stuff. It is not like we rewrite because code is crap. We have like 30 percent of test coverage because that is part that changed only a bit.
Post reply on HN