Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

321–330 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#321

Post by dllthomas »

ericdykstra wrote: post #210
tbrownaw wrote: post #121

Earlier quoted context omitted.

the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely u…

I really don't believe any study or meta-study could come close to being able to suss out the nuance of when TDD may provide an advantage and when it doesn't. I'd rather just trust programmers to consider what approach works best for their problem and mindset and go from there. I personally don't TDD most things, but it's a tool I have available and I bring it out when a situation arises.

I think we could, but it'd be very expensive (multiple large and careful studies) and only possibly worth it.

> I'd rather just trust programmers to consider what approach works best for their problem and mindset and go from there.

People are often surprisingly good at fooling themselves. I'd rather have actual empirical validation. And also a pony.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#322

Post by dllthomas »

zorked wrote: post #143
bjelkeman-again wrote: post #132

Earlier quoted context omitted.

> provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?

You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).

I've heard that tests written specifically to drive up coverage metrics aren't much better than no tests.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#323

Post by itsdrewmiller »

noxToken wrote: post #291
hirzel wrote: post #279

Earlier quoted context omitted.

Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?

Yes. Some people do not know how to write extensible software. They solve the issue at hand without any regard to big picture, so when something needs to be added (like test) the code needs to be heavily refactored to accommodate the change. As others have stated, this usually(!) isn't a big deal when the project is small with a single contributor. However, writing the tests last without any forethought to what the t…

If true, wouldn't this be detectable in academic studies? It's easy to write untestable code for small problems, too.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#324

Post by jdlshore »

rainforest wrote: post #260

The use of students in SE research is a hot topic, see, for example Fietelson's review: https://arxiv.org/abs/1512.08409 . Practitioners have a problem recruiting subjects. There is often a tradeoff between applying more rigorous experiment design and using convenience sampling (students) versus sacrificing controlled environments (so that professionals would actually join the study). It's easy to condemn work like t…

Your argument reminds me of the joke about the man searching for his keys under the streetlamp.

"A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, 'this is where the light is.'" [https://en.wikipedia.org/wiki/Streetlight_effect]

I agree that this is a well-designed study given its constraints. And it's admirable that it's a replication study.

That doesn't change the fact that it's largely irrelevant to professionals. It doesn't test the claims made by TDD proponents (TDD leads to better design, reduces long-term maintenance, allows for team coordination, etc.), nor does it address any of the interesting questions about TDD:

* Is TDD more effective in a professional setting than commonly-used alternatives?

* Is a mock-heavy approach to TDD more effective than a mock-light approach?

* Do people using TDD refactor their code more or less than people using a different but equally rigorous approach?

* Is the code done with TDD more maintainable than code done rigorously in another way?

* Is TDD easier or harder to sustain than equivalently-effective alternatives?

As a study, it's fine, if only of interest to academics. The problem isn't the study. It's the credulous response on the part of industry developers who then turn the false authority of the study into statements like "TDD doesn't lead to higher quality or productivity."

Re: An External Replication on the Effects of Test-driven Development [pdf]

#325

Post by jdlshore »

rainforest wrote: post #260

The use of students in SE research is a hot topic, see, for example Fietelson's review: https://arxiv.org/abs/1512.08409 . Practitioners have a problem recruiting subjects. There is often a tradeoff between applying more rigorous experiment design and using convenience sampling (students) versus sacrificing controlled environments (so that professionals would actually join the study). It's easy to condemn work like t…

I skimmed the Fietelson's paper you linked and this jumped out at me:

"Students should generally not be used in studies that depend on specific expertise which requires significant experience and a long learning curve to achieve, or in studies of professional practices. Such studies are best performed by observing and interviewing professionals, not by controlled experiments."

This seems directly relevant to this TDD study. Every one of these contraindications are true in this case.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#326

Post by beambot »

adwn wrote: post #250
beambot wrote: post #243

Earlier quoted context omitted.

Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Results are based on statistical relevance subject to the sampling (and their biases). This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, me…

> Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Neither jdlshore nor I were talking about "proof" or "truth" in the mathematical sense. > Results are based on statistical relevance subject to the sampling (and their biases). The point was that this study doesn't have sufficient statistical relevance to give any ev…

> [...] whether TDD is effective in general.

If you read the actual article (even just the abstract), you would realize that evaluating TDD in a general professional settings was neither the goal nor the conclusion of the authors. You are arguing a straw man and hoping to make inferences that are unsupported by the paper's claims. The authors do a good job of scientific communication about their methods, results, and limitations. This is good practice for scientific communications. Don't think it's sufficiently generalizable? Fine, feel free to expand upon their work. That's how science functions.

("truth", "prove", and "rigor" were verbatim, primary components of jdlshore's comment. The two former words have very specific scientific meaning.)

Re: An External Replication on the Effects of Test-driven Development [pdf]

#327

Post by smallnamespace »

vacri wrote: post #274
smallnamespace wrote: post #170

Earlier quoted context omitted.

This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.

Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.

I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#328

Post by willtim »

icebraining wrote: post #181
willtim wrote: post #113

Earlier quoted context omitted.

Processes like TDD are most often applied/enforced with inexperienced developers in mind.

I've never seen Kent Beck make that claim, have you?

Has Kent Beck been pushing for TDD at Facebook, or are there too many experienced devs?

Experienced devs do what works best for them, which may be TDD but probably in most cases not.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#329

Post by lgunsch »

rubber_duck wrote: post #188
blazespin wrote: post #177

Earlier quoted context omitted.

The study is huge huge HUGELY flawed. TDD only comes into value when you start having to refactor extremely large projects that you don't understand. You need the tests in order to refactor with confidence.

You can have tests without TDD ? TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests. Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the…

> TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests.

This is not how TDD works, and I can definitely see how writing code in that manner wouldn't be very productive at all. I also used to think that was how TDD was done until I read how it's really supposed to work by experts.

You iteratively build up code in a small tight cyclic manner. You do not write 100% of your test in 1 go, and you do not write 100% of the code in one go. You are also missing a step. TDD is a 3 step process, red-green-refactor, and the last missing step is refactoring. You start off with a small atomic behaviour you need in your API/product. You then make a test for that behaviour. The behaviour is not related to a particular class or method. Perhaps it is 1 class by coincidence , but maybe its 3 or 4 classes together. A good example of this would be if I had a method that dealt with multiple items. I would start off by constructing my test to use 1 item, and and the code to deal with 1 item. Then the next iteration maybe a list of 1 item in the test and code. After that, maybe I'd move to multiple items in a list.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#330

Post by ivan_gammel »

platz wrote: post #313
ivan_gammel wrote: post #286

Earlier quoted context omitted.

Well, I practiced TDD for no less than 10 years, and its benefits are still not obvious, more like something that needs to be continuously reviewed in the context of the current work. Once you've understood all the design patterns that TDD enforces, you don't need TDD to use them. You just think how you _would_ test your code and defer test implementation for future.

Why not just learn the patterns directly?

:) Who said they should not be learned directly? TDD is just a good demonstration of their use.
Post reply on HN