Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

241–250 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#241

Post by jacquesm »

jaggederest wrote: post #183
vacri wrote: post #145

Earlier quoted context omitted.

Out of curiousity, how many subjects would you say are necessary to start being statistically valid?

Kind of the wrong question. It's more about qualitative sample selection and the expected effect size than it is the number of subjects. Based on my admittedly poor understanding, I would expect to see a sample size of around 70 to have the power to detect a small effect in a single-tailed T test. I would also expect to have the samples selected and balanced carefully, preferably from the target group (i.e. experienc…

Especially not because the major benefit of TDD hits much later in the cycle, during refactoring a large chunk of code or when doing some major surgery on the whole project (and let's hope the tests are at the interface level).

Re: An External Replication on the Effects of Test-driven Development [pdf]

#243

Post by beambot »

adwn wrote: post #195
beambot wrote: post #156

Earlier quoted context omitted.

> The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. That's where you're wrong... even with "careful statistics", the point of sample-based studies is to disprove the null hypothesis with some confidence level . There is no requirement that it be 100% confident. In fact, with p (See, it's easy to get this wrong!)

Although I believe that you're technically correct, your objection does not refute jdlshore's central point.

Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Results are based on statistical relevance subject to the sampling (and their biases).

This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, merely the scope. In this case, what the researchers actually found: "At confidence interval, TDD doesn't work for white, male graduate students at University working on problem. Generalize at your own peril." But that's a shitty headline.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#244

Post by dang »

We changed the URL from http://neverworkintheory.org/2016/10/05/test-driven-developm..., which points to this.

When the topic is controversial and the paper is not so specialized that only a few people here can understand it, changing the URL to that of the paper tends to help make a discussion more substantial. Especially when the blog post is more of a gloss on the paper than an in-depth commentary on it.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#245

Post by rubber_duck »

crdoconnor wrote: post #237
rubber_duck wrote: post #188

Earlier quoted context omitted.

You can have tests without TDD ? TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests. Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the…

>You can have tests without TDD ? I've worked on several projects without this and the tests done without TDD tend to be of higher quality. I noticed a common anti-pattern of "write the code, run the code, copy the output of the code, paste it into a test and write an assert to check that the output was precisely what came out". This was brittle, it killed the self-documenting aspect of the tests and it often conceal…

>I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in.

How do you know you made API mistakes from writing unit tests ? API mistakes become apparent when you integrate stuff and use the API in conjunction with other things (using API in isolated scenarios like unit tests is not really insightful you can figure out those flaws just by looking at the API). This is my primary criticism of TDD - you write tests around the initial bad API and you end up keeping that bad design because you already spent so much time testing that. Implementation errors caught by unit tests are cheaper to fix afterwards then design errors.

My approach is hack together a POC -> refactor that and write tests. If you discard the POC then TDD makes sense to me.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#246

Post by dang »

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

> It doesn't justify the sensational title here on HN.

We've changed the title along with the url. Please see https://news.ycombinator.com/item?id=12742120. The submitted title was “TDD has little or no impact on development time or code quality” (including the quotes).

Re: An External Replication on the Effects of Test-driven Development [pdf]

#247

Post by bnegreve »

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

To be fair, the abtract says "The results failed to support the claims." and the conclusion says "We recommend future studies to survey the tasks used in experiments evaluating TDD, and assess them with respect to the treatments.".

This is not a claim that TDD is useless.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#248

Post by regularfry »

geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

The "lightbulb" moment for me was doing what Kent Beck suggests, and writing down the list of tests I "knew" I needed to write, before writing any tests. That's a very similar step to diving into writing the code first, "knowing" what it ought to look like, and the ideas feeding into either step would be the same.

Writing a list of tests first is quicker, though, and working through the list, updating it as I learn more, tells me that my initial ideas were wrong often enough that I'm almost convinced that the true secret of TDD isn't the tests at all. It's that checklist, and the purely mechanical process of working through it, checking them off one-by-one and fixing it up as I go along.

To relate this to your situation, I often find that I'll only get started with a fairly fuzzy idea, and the checklist will be short. I can still expand it as I go, knowing that I'm working on as solid foundations as I can given the constraints.

I think there's a danger in writing off practices like TDD with "if it works for you..." because it avoids discussion of why it works for some people and not others. That in turn gives people license to ignore it simply because it's unfamiliar, and the brain doesn't like unfamiliar things by default. That's a shame, because for some of those people they'll be stuck in local optima, writing off the one thing which could break them out of it.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#249

Post by crdoconnor »

geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

>Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc.

I generally find it better to engage in "top-down" development - specification->tests->high level code->low level code as it's cheaper to fix mistakes at higher levels first.

If I'm doing purely experimental code though - where I'm not sure of what I even want or what the output should be, tests are just a waste of time.

It also loses its effectiveness the more declarative the code is. If I'm essentially tweaking configuration or HTML there's no point.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#250

Post by adwn »

beambot wrote: post #243
adwn wrote: post #195

Earlier quoted context omitted.

Although I believe that you're technically correct, your objection does not refute jdlshore's central point.

Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Results are based on statistical relevance subject to the sampling (and their biases). This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, me…

> Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense.

Neither jdlshore nor I were talking about "proof" or "truth" in the mathematical sense.

> Results are based on statistical relevance subject to the sampling (and their biases).

The point was that this study doesn't have sufficient statistical relevance to give any evidence whether TDD is effective in general. It doesn't matter if this study gives any evidence whether TDD is effective when used by graduate students working on toy problems, because that was not the intention of the study (besides, nobody cares about this highly specific case).

Post reply on HN