New programmers will read "studies" like this and will decide to write tests "someday later". I really hate impact this "study" brings. And I agree with every point of @jdlshore comment.
An External Replication on the Effects of Test-driven Development [pdf]
191–200 of 332 posts
Re: An External Replication on the Effects of Test-driven Development [pdf]
#192Earlier quoted context omitted.
If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…
Studies with small 'n' are feeder studies, exploring large amounts of problem space quickly. When patterns emerge, they can be retested or expanded upon. Equating a somewhat flawed study to 'a persuasive essay' is wholly disingenuous. As is claiming n=20 to be 'tiny'. It's small, sure, but not ridiculously small. This kind of study isn't a final nail in the coffin, but a data point to add to the discussion.
The use of students for this purpose reminds me of a story about a man who designs a flying machine.
After months of calculations and material research, he determines that only if he builds his machine out of birch will it be light enough to fly. According to his calculations, nothing else will work.
Days later, his best friend is visiting him in the hospital. He carries over a piece of the broken machine. With a puzzled look on his face, the friend asks, "Why did you build it out of pine when you knew you needed birch?"
"Because I didn't have any birch."
Re: An External Replication on the Effects of Test-driven Development [pdf]
#193After a quick read on the metrics section, it seems the quality is measured in terms of adherence to user stories implemented as a set of behavior tests. There seems to be no assessment on code maintainability and looks like a flaw in the study, as it would model a short lived codebase and not one that undergoes several maintenance cycles.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#194This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…
* The code-base was new, simple, and subsequently thrown away.
* There is no way to scientifically measure absolute code quality.
There are numerous subjective indicators (tabs vs. spaces) and the objective indicators only really become apparent once you are dealing with more than one component/service.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#195Earlier quoted context omitted.
If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…
> The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. That's where you're wrong... even with "careful statistics", the point of sample-based studies is to disprove the null hypothesis with some confidence level . There is no requirement that it be 100% confident. In fact, with p (See, it's easy to get this wrong!)
Although I believe that you're technically correct, your objection does not refute jdlshore's central point.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#196Earlier quoted context omitted.
> Untested code is nigh impossible to refactor, so nobody ever does, and the end result is usually piles of hacks upon piles of hacks. The mistake is in creating unrefactorable code. TDD may be a possible solution if done correctly but there are other ways to skin that cat.
Sure, but I have yet to find a better solution than TDD for big teams where team members come and go constantly.
"Branches aren't merged without peer approval" has sufficed plenty in my experience.
Whether people code tests before thinking interfaces, before writing a prototype, before implementing, during the inevitable interface rewriting, after coding, after manual verification that it seems to work, or right before submitting the branch to review, doesn't matter as long as someone on the team looks over and sees that "yup, here are tests and they seem to cover the important parts, the interface makes sense, and the documentation is useful".
Then people can do TDD, TLD, TWD or whatever they personally feel most productive with. Developers being happy and feeling in control of their own work does more for quality than enforcing a shared philosophy.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#197The title suggests that TDD has little or no impact on dev time or code quality at all. The research shows no significant difference between TDD and iterative test-last (ITL) development. Could the title be updated? To show that it is a comparison of TDD vs ITL/TLD.
This thread feels like a study showing that HNers don't read the articles.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#198It always both amuses and saddens me how people will eagerly write more tests than actual code, but refuse to use a strongly typed language. The compiler is my test harness.
The compiler can test the validity of your code, not it's behavior. Only the most trivial bugs can be caught this way.
Re: An External Replication on the Effects of Test-driven Development [pdf]
#199As a TDD advocate, and assuming this study has any scientific validity, this is actually good news! There's a very common claim that TDD makes you less productive. It's good to have some study to oppose this claim.
I think methodically you'd need equivalence testing for that. Your hypothesis would be that productivity is equal (enough) and you could then discuss the additional benefits of TDD.
I can't remember if I read a study like this for TDD but equivalence testing is fairly underused outside of pharma/medicine (it's often even called bioequivalence) where the test usually shows similar enough effects and the extra benefits are cost savings (for generics).
Re: An External Replication on the Effects of Test-driven Development [pdf]
#200"TDD disproven by people who have no idea how to practice it, or have the ability to grok the longterm benefits."