This is an editorialised title. The blog posting is a boring "Test Driven Development". The blog posting and the paper that it fronts has a conclusion that no significant difference between TDD and iterative test-last (ITL) development, which is quite a bit different from TDD has little or no impact on development time or code quality
An External Replication on the Effects of Test-driven Development [pdf]
111–120 of 332 posts
Re: An External Replication on the Effects of Test-driven Development [pdf]
#112Earlier quoted context omitted.
This is why designing the interface in a language with a powerful type system provides the same quoted benefit of TDD. It helps you think about the interface before you move to implementation. In my experience, using a type system to do this requires much less effort and refactoring.
I am a big fan of "testing via types". Unit tests can demonstrate that my code is correct; it can say nothing about how my code is used by others. In contrast, a type system can extend those protections downstream. (Yes, yes, unless you go way off the deep end of dependent types, you still need tests. But a powerful type system can systemically prevent a huge number of very common bugs.)
My take it on it is that it can remove most of the boring bugs you would get in a less strongly-typed language and leaves you open to the interesting bugs (eg, logic bugs).
As for dependent types, I'm not sure that even then you would be able to ensure that, say, your cache is correctly invalidated.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#113This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…
Processes like TDD are most often applied/enforced with inexperienced developers in mind.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#114The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…
Potentially contentious opinion, but one that's been echoed through our halls - Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled. It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably…
Not to mention that there is a lot of stuff, especially for UI, that is either hard or impossible to write tests for...
Re: An External Replication on the Effects of Test-driven Development [pdf]
#115Earlier quoted context omitted.
It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…
Exactly this; I find too many HN comments to be critical in a non-constructive manner.
The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#116If the claim in the paper is true:
TDD = Same time + same quality + feel better.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#117This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…
Whereas the same conditions you listed existed in the link I just found on Cleanroom method from 80's with clear evidence the method worked to produce low defect software. The weaknesses plus resulting quality just added further support.
http://infohost.nmt.edu/~al/cseet-paper.html
EDIT: This is now it's own submission below if anyone wants to discuss it there.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#118The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…
Potentially contentious opinion, but one that's been echoed through our halls - Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled. It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably…
I've actually found that most QA people I've worked with were good at deciding what to test but were awful at writing automated tests. Of course that's purely anecdotal and based on a very small sample size.
I like to believe that I sometimes will forget to test for edge cases, but have mastered how to write tests. I don't think that how to write tests is easily acquired by any descend programmer. However, you need to learn a few patterns you otherwise won't need.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#119This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…
It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…
I completely disagree. The reason we care about these studies is to make decisions. If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny.
It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive.
If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark.
It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students.
And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#120This is a misleading title and conclusion. The study showed a huge benefit of TDD over Waterfall, and it is only when compared to ITL that it was found to not be better. But moreover, I think it's important to understand why Beck pushed for TDD. TDD is like saying "I'm going to floss before I brush every time, no matter what." But, when people don't do TDD they typically aren't all saying "I'm going to brush and flos…
Waterfall is a straw man.
Not only that... when you test the efficacy of medical interventions the gold standard to strive for[1] is not whether the new intervention is better than placebo, it's whether it's better than $CURRENT_BEST_KNOWN_INTERVENTION. I suggest we should be aiming for a similar standard in testing software engineering methodology.
I think it would be very hard to argue that Waterfall ~= $CURRENT_BEST_KNOWN_METHODOLOGY.
[1] Of course, this isn't usually what happens in practice when pharmaceutical companies are doing their own testing, but it's what should happen if you actually care about efficacy and not just PR/sales.