Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

281–290 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#281

Post by pmontra »

mseebach wrote: post #108
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

TDD means that you write tests first, see them fail, write the code, see the tests pass. Writing the code first and then the tests is not TDD. I didn't find much value in tests first, especially because sometimes I really dont don't have any idea of what tests to write and I have to sketch some code anyway. I tend to write some code, then its tests and use them to debug my code. I save the time spent to write some tests that will be irrelevant to the code I eventually end up writing. In very few cases, let's say twice per year, I have to solve very clear algorithmic problems with very well defined inputs and outputs. Example: process the elements of an array and return another array. Then I write the tests first.

Anyway, tests first or tests later both are good. No tests is a nightmare as a couple of projects I've just inherited are reminding me today.

Edit: typoes

Re: An External Replication on the Effects of Test-driven Development [pdf]

#282

Post by hueving »

apatters wrote: post #127
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

Have you heard the phrase "garbage in, garbage out?"

Re: An External Replication on the Effects of Test-driven Development [pdf]

#283

Post by candiodari »

jaunkst wrote: post #43

TDD is king when refactoring, or proving an algorithm. You have a tests to confirm the output, and near realtime feedback that you assumptions are correct. The rest is obvious. Mission critical component TDD, complicated refactor TDD, algorithm you need to validated TDD. Anything else write the code and get a peer review.

No amount of tests can prove that an algorithm is correct. At best, they prove that an algorithm works in a particular case.

And generally, I would say that the more interesting sorts of tests (fuzz testing, large-scale system testing) are extremely unpopular with software engineers because "they suddenly fail without reason". Not quite as unpopular as actual proof that an algorithm works, like implementing it in coq for instance, but very unpopular.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#284

Post by onion2k »

rlpb wrote: post #229
onion2k wrote: post #126

Earlier quoted context omitted.

I have a somewhat different experience. I've found TDD is useful on even tiny projects because using TDD forces you to write better code. It's really hard to reliably test things that use "magic" and side effects that affect parts of your code they shouldn't really be affecting, and really easy to test things that have well-defined and properly documented interfaces that only do one thing. Consequently writing tests,…

> Consequently writing tests, even if you never actually run them, makes your software better. It could be argued that an experienced programmer doesn't need TDD to write the same code that TDD could have produced.

I've only been a developer for 20 years so I'm not at that point yet.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#285

Post by johan_larson »

pkolaczk wrote: post #264
johan_larson wrote: post #162

The questions worth asking about techniques like TDD are "What problems does it fix?" and "What problems does it introduce?" I would expect a determined attempt at TDD to solve the "no tests" problem, because it is so utterly insistent on tests. It should also solve the "don't know how to start" problem, because it de-emphasizes planning and design in favor of just jumping in; you write the tests, and then you do the…

" It should also solve the "don't know how to start" problem, because it de-emphasizes planning and design in favor of just jumping in; you write the tests, and then you do the bare minimum to make them pass" It doesn't SOLVE this problem. It only pushes it in time, making it actually worse, because you're wasting time on stupid tests instead of actively researching a solution. No amount of tests is going to help you…

I think TDD deserves a bit more credit than that. Building a simple solution for part of the problem can be credited as exploring the problem space. The same can be said for extending it to address more of the problem; that's exploration too.

But I fear this incremental approach is going to produce a very baroque solution that is going to have to be rewritten completely once the bell goes "bing" and the programmer actually understand the underlying problem well enough to produce a clean solution.

I think the larger problem with TDD is that there are at least four parts to software design, and TDD bets it all on two of them. There's requirements analysis, architectural design, construction, and finally testing. TDD is really all about the construction and testing bits. It doesn't address requirements analysis at all, and it doesn't seem to want to do architectural design, it just constructs and tests with great passion. It's imbalanced.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#286

Post by ivan_gammel »

spatulon wrote: post #278
ivan_gammel wrote: post #253

Earlier quoted context omitted.

>or any style of automated tests, really, you don't really have to be driven by them Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in j…

This benefit of test-driven design is not obvious (at least it wasn't for me) until you really experience it. Several years back I wasn't aware of the concept of dependency injection, but independently discovered it while trying to unit test some error logging functionality. Previously I'd always used a global/singleton logger object. That always felt wrong, but I always assumed it was necessary until writing unit te…

Well, I practiced TDD for no less than 10 years, and its benefits are still not obvious, more like something that needs to be continuously reviewed in the context of the current work. Once you've understood all the design patterns that TDD enforces, you don't need TDD to use them. You just think how you _would_ test your code and defer test implementation for future.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#287

Post by TheOtherHobbes »

StreamBright wrote: post #226
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

I agree with you on this. The study is irrelevant from the statistical point of view. However, there are other considerations with TDD, probably on a more theoretical level that: * There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?) * There is one thing common for sure with all of the bugs you catch in production, they a…

Car manufacturers model car bodies mathematically so they have a good idea how a body will behave before they build it.

This hints at why the philosophy of TDD is not as helpful as it seems: it confuses modelling/analysis with implementation.

Your testing methodology can be perfectly implemented, but if your problem analysis is wrong your software will still be useless and broken.

TDD would be more interesting if it understood the distinction between specific behavioural expectations (easy to test for, but incomplete), generic applicability and model robustness (much harder to test for, but more complete), and complete formal correctness (often unreachable, but always an interesting goal.)

TDD is better than nothing. If all you can manage is a set of behavioural tests, that should still improve reliability.

But it's a big leap from there to the suggestion that if you define your spec as a series of behavioural tests and your code passes them all, you have a full and correct solution for the initial requirement.

That is just plain wrong, and in untrained hands it can be dangerous.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#288

Post by johnlbevan2 »

TL;DR

Conclusion: "TDD does not affect testing effort, software external quality, and developers’ productivity"

However, per jdlshore's comment (https://news.ycombinator.com/item?id=12740978), test parameters weren't suitable for any meaningful conclusions to be drawn.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#289

Post by thedufer »

icebraining wrote: post #179
simula67 wrote: post #133

Earlier quoted context omitted.

One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.

Nope, there are also studies (done by Microsoft and IBM) indicating that TDD helps: https://www.infoq.com/news/2009/03/TDD-Improves-Quality

It isn't terribly clear from the abstract and I'm not going to pay for the paper, but it sounds like this is really a comparison of tests vs no tests. That has nothing to say about the practice of TDD.

This is why this newer study is interesting; it compares to TLD, which allows you to say something about TDD rather than just about testing.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#290

Post by Udik »

hueving wrote: post #282
apatters wrote: post #127

Earlier quoted context omitted.

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

Have you heard the phrase "garbage in, garbage out?"

Referred to the study or to the commenters?
Post reply on HN