Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

141–150 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#141

Post by gohrt »

tspike wrote: post #14

My experience has been that TDD is worthwhile when working with notoriously slippery whack-a-mole functions like handling time or money. The time saved by catching regressions vastly outweighs the time taken to implement the tests. In contrast, TDD has been a waste of time for me for UI-based work, as the effort needed to properly expose the functionality under test is too great and the requirements and design change…

You are saying "TDD" but your comment reads the same if you just say "tests in general"

Re: An External Replication on the Effects of Test-driven Development [pdf]

#143

Post by zorked »

bjelkeman-again wrote: post #132
tbrownaw wrote: post #121

Earlier quoted context omitted.

the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely u…

> provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?

You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).

Re: An External Replication on the Effects of Test-driven Development [pdf]

#144

Post by vacri »

jdlshore wrote: post #137
tlarkworthy wrote: post #131

Earlier quoted context omitted.

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…

Studies with small 'n' are feeder studies, exploring large amounts of problem space quickly. When patterns emerge, they can be retested or expanded upon.

Equating a somewhat flawed study to 'a persuasive essay' is wholly disingenuous. As is claiming n=20 to be 'tiny'. It's small, sure, but not ridiculously small.

This kind of study isn't a final nail in the coffin, but a data point to add to the discussion.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#145

Post by vacri »

r00fus wrote: post #136
tlarkworthy wrote: post #131

Earlier quoted context omitted.

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.

Out of curiousity, how many subjects would you say are necessary to start being statistically valid?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#146

Post by zby »

geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

For me it also depends on the task - I never do a strict TDD - but often I like to start with some tests.

Sometimes it is hard to write the tests, but it is easy to see if your code works - sometimes it is easy to write the tests, but not the code.

I would call it 'do first what is easier'.

But it is important to write the tests quickly after you wrote the piece of code - so that you don't forget something.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#147

Post by SatvikBeri »

tlarkworthy wrote: post #131
tikhonj wrote: post #119

Earlier quoted context omitted.

I completely disagree. The reason we care about these studies is to make decisions . If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in ever…

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

There are really two types of studies: exploration and validation. Exploratory studies on small samples are still useful for refining hypotheses and figuring out whether they deserve more resources, but shouldn't be seen as significant evidence towards a conclusion–they are generally _less_ valid than opinions from people experienced in the field.

In contrast, validating studies _should_ have large enough sample sizes to be statistically significant, precommit to sharing results in order to avoid publication bias, have well-validated experimental designs, etc.

The problem is that these two are often conflated, so most people either blindly trust studies or put no faith in them whatsoever.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#148

Post by EtienneK »

jcoffland wrote: post #99
donw wrote: post #68

Am I correct in reading that they performed this experiment only for two days, and entirely with graduate students? If so, they have missed the point of TDD. In the short term, TDD probably doesn't make a difference, one way or another. But software as a business is not a short-term game. I would love to see a study where the participants are, over a period of six months, given the same series of features (including…

> Untested code is nigh impossible to refactor, so nobody ever does, and the end result is usually piles of hacks upon piles of hacks. The mistake is in creating unrefactorable code. TDD may be a possible solution if done correctly but there are other ways to skin that cat.

Sure, but I have yet to find a better solution than TDD for big teams where team members come and go constantly.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#150

Post by mehwoot »

EdSharkey wrote: post #100

The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…

The study is about the effects of "test first" vs "test later", not "writing tests" vs "not having tests".
Post reply on HN