An External Replication on the Effects of Test-driven Development [pdf] • Hacker News Board

Re: An External Replication on the Effects of Test-driven Development [pdf]

#131

Post by tlarkworthy » Wed, Oct 19, 2016, 5:29 AM UTC

tikhonj wrote: post #119

hackuser wrote: post #98

Earlier quoted context omitted.

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

I completely disagree. The reason we care about these studies is to make decisions . If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in ever…

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication.

There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing every study as irrelevant due to lack of rigor is wasteful.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#132

Post by bjelkeman-again » Wed, Oct 19, 2016, 5:29 AM UTC

tbrownaw wrote: post #121

ericdykstra wrote: post #106

Earlier quoted context omitted.

The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story…

the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely u…

> provide no additional benefit over anything else that gets the tests written.

Isn't this enough though? The tests gets written, which probably was the main point in the first place?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#133

Post by simula67 » Wed, Oct 19, 2016, 5:30 AM UTC

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#134

Post by Udik » Wed, Oct 19, 2016, 5:31 AM UTC

madeofpalk wrote: post #76

bunderbunder wrote: post #25

Earlier quoted context omitted.

For me it's a highly situation-dependent thing. Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth betw…

Yes. When doing UI code (with something like React) it's next to impossible to do TDD - you can't know the structure until you actually implement it. If I'm fixing a bug in some sort of non-ui code, I often find TDD to be very helpful. Create a test case that demonstrates that failure, then fix the implementation to make the test pass.

I'm not sure that testing a bug, even before writing the code that fixes it, qualifies as TDD.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#135

Post by kodfodrasz » Wed, Oct 19, 2016, 5:37 AM UTC

hirsin wrote: post #107

EdSharkey wrote: post #100

The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…

Potentially contentious opinion, but one that's been echoed through our halls - Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled. It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably…

I have experienced the opposite. After the dedicated QA team was disbanded and repurposed to development, where every developer had to write tests for someone else's code, the code and designs started to become more testable, the tests eventually became simpler to maintain and understand. (the project was and embedded system)

The "We" and "Them" distinctions made people in different teams and different roles ignore the needs of the others. The change brought a culture change and also gave a view on the needs of the other side. This could make it possible that for the first time the development of tests and features could really be done in parallel (officially that was the methodology before, but never really worked out well, because the testability considerations were usually ignored at design time, and the docs were lagging, because of the bad bandwagoning culture. With the mixed team where everyone was treated as an equal these problems dissolved surprisingly quickly (in less than half a year)).

So my point is having dedicated testers can give base for bad culture which hurts the product and the company. Having everyone do the same job with regards to development and testing is better.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#136

Post by r00fus » Wed, Oct 19, 2016, 5:39 AM UTC

tlarkworthy wrote: post #131

tikhonj wrote: post #119

Earlier quoted context omitted.

I completely disagree. The reason we care about these studies is to make decisions . If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in ever…

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#137

Post by jdlshore » Wed, Oct 19, 2016, 5:42 AM UTC

tlarkworthy wrote: post #131

tikhonj wrote: post #119

Earlier quoted context omitted.

I completely disagree. The reason we care about these studies is to make decisions . If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in ever…

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority?

The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anything about TDD in a professional setting. For us professionals, it's noise. Yet it's sitting at the #1 spot on HN, fulfilling people's preconceptions.

Also, the replication crisis [1] would disagree with you.

[1] https://en.wikipedia.org/wiki/Replication_crisis

Re: An External Replication on the Effects of Test-driven Development [pdf]

#138

Post by jaredklewis » Wed, Oct 19, 2016, 5:44 AM UTC

tlarkworthy wrote: post #131

tikhonj wrote: post #119

Earlier quoted context omitted.

I completely disagree. The reason we care about these studies is to make decisions . If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in ever…

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

Well, if we are discussing the allocation of resources, I for one would much prefer fewer, more rigorous studies than a large volume of studies with mediocre methodology and sample sizes. Such studies carry little more weight than one's own intuitions.

I do feel that small studies like this one serve a very important purpose: to let scientists hone their experiments. Most any large study should first be attempted as a small study to work out any kinks.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#139

Post by djsumdog » Wed, Oct 19, 2016, 5:45 AM UTC

ericdykstra wrote: post #106

hackuser wrote: post #98

Earlier quoted context omitted.

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story…

You know, this study and others don't take about long term maintainability or brekability.

In fact, isn't this a good thing? If it has little impact on development time, we should totally be doing it. That way, we have tests. We have them from the start. We met the conditions we wanted instead of guessing conditions at the end.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#140

Post by staticelf » Wed, Oct 19, 2016, 5:48 AM UTC

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

Yes it's flawed, but so is development methodologies in general.

The truth is that it depends on the team writing the code not how you write it. While consistency and a method may help to organize the group it doesn't really say anything about any quality what so ever.