An External Replication on the Effects of Test-driven Development [pdf] • Hacker News Board

Re: An External Replication on the Effects of Test-driven Development [pdf]

#181

Post by icebraining » Wed, Oct 19, 2016, 6:58 AM UTC

willtim wrote: post #113

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

Processes like TDD are most often applied/enforced with inexperienced developers in mind.

I've never seen Kent Beck make that claim, have you?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#182

Post by avarun » Wed, Oct 19, 2016, 7:00 AM UTC

kriro wrote: post #176

jdlshore wrote: post #137

Earlier quoted context omitted.

If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…

I'm not sure it's pure noise for professionals. These students eventually turn into professionals so there's probably some relationship (at least that would be my hypothesis). Regarding the replication crisis...this is actually a replication and there's actually more replication for TDD than for most topics I read about.

Again, as a convenience sample with a tiny sample size, there's no conclusion we can even draw about students, let alone professionals.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#183

Post by jaggederest » Wed, Oct 19, 2016, 7:03 AM UTC

vacri wrote: post #145

r00fus wrote: post #136

Earlier quoted context omitted.

Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.

Out of curiousity, how many subjects would you say are necessary to start being statistically valid?

Kind of the wrong question. It's more about qualitative sample selection and the expected effect size than it is the number of subjects.

Based on my admittedly poor understanding, I would expect to see a sample size of around 70 to have the power to detect a small effect in a single-tailed T test. I would also expect to have the samples selected and balanced carefully, preferably from the target group (i.e. experienced professional programmers in the language at hand)

From what I can tell, for an effect to be detected at least 80% of the time in a sample of 20 people, you'd need the effect to be >60%. I would definitely not expect that a short term project would show 60% improvement when employing similar levels of testing, changing only either before or after writing the code under test.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#184

Post by barking » Wed, Oct 19, 2016, 7:04 AM UTC

olalonde wrote: post #77

bunderbunder wrote: post #25

Earlier quoted context omitted.

For me it's a highly situation-dependent thing. Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth betw…

> Sometimes writing tests first helps me think about the high-level design of my code. I'm personally a fan of "readme driven development" ( https://news.ycombinator.com/item?id=1627246 ). It's a nice compromise between tons of upfront planning and nothing at all.

That's an interesting discussion and article. I sort of do this already for new features. You really feel the power of it, when after writing a lot of stuff you realise that there's a hitch, rewrite from earlier and shudder at the thought of the mess and wasted time if code had been written. In its final state the document becomes a series of steps that I can check off, which is helpful in a motivational sense.

I had thought of involving users in the design of new features via the support forum but I'm afraid of how that might turn out. A readme type document would probably be about as good a way of doing it as any though.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#185

Post by zamalek » Wed, Oct 19, 2016, 7:05 AM UTC

embwbam wrote: post #88

paulddraper wrote: post #85

Earlier quoted context omitted.

TDD vs. test-after-code is a small distinction. 80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions. So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).…

This is why designing the interface in a language with a powerful type system provides the same quoted benefit of TDD. It helps you think about the interface before you move to implementation. In my experience, using a type system to do this requires much less effort and refactoring.

Additionally, strict TDD teaches you lessons. Once you've used it enough, chances are that you'll be making better interfaces even if you code first.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#186

Post by EugeneOZ » Wed, Oct 19, 2016, 7:06 AM UTC

hackuser wrote: post #98

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

And words of that professor don't fix anything, really. I absolutely, 101% agree with comment you are replying.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#187

Post by dasmoth » Wed, Oct 19, 2016, 7:09 AM UTC

ajmurmann wrote: post #118

hirsin wrote: post #107

Earlier quoted context omitted.

Potentially contentious opinion, but one that's been echoed through our halls - Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled. It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably…

I've actually found that most QA people I've worked with were good at deciding what to test but were awful at writing automated tests. Of course that's purely anecdotal and based on a very small sample size. I like to believe that I sometimes will forget to test for edge cases, but have mastered how to write tests. I don't think that how to write tests is easily acquired by any descend programmer. However, you need t…

The real strength of QA people can be manual testing. There's a local maximum of "meets the stated requirements but horrid to actually use", and a singular focus on test automation seems to push projects in that direction.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#188

Post by rubber_duck » Wed, Oct 19, 2016, 7:10 AM UTC

blazespin wrote: post #177

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

The study is huge huge HUGELY flawed. TDD only comes into value when you start having to refactor extremely large projects that you don't understand. You need the tests in order to refactor with confidence.

You can have tests without TDD ?

TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests.

Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the insight gained. Then I spec out the behavior with tests. TDD assumes you have the design/spec right from the start and all you need to do is write the implementation - very little of my work falls in to that category - perhaps it's different for others.

One segment where I found TDD useful is writing story level E2E tests that spec out the requirements before the code is written - this is the least ambiguous way to spec the problem I've found - the downside being that the person doing the spec needs to know how to write E2E tests.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#189

Post by kodfodrasz » Wed, Oct 19, 2016, 7:12 AM UTC

hvidgaard wrote: post #168

kodfodrasz wrote: post #135

Earlier quoted context omitted.

I have experienced the opposite. After the dedicated QA team was disbanded and repurposed to development, where every developer had to write tests for someone else's code, the code and designs started to become more testable, the tests eventually became simpler to maintain and understand. (the project was and embedded system) The "We" and "Them" distinctions made people in different teams and different roles ignore t…

There is no silver bullet here. For some teams it makes sense to integrate development and testing. For some teams it makes sense to have dedicated QA people. For some teams a different constellation is optimal. I know developers that are brilliant at the big picture, finishing the implementation however was lacking. I also know developers that cannot get the major architecture right, but they can finish tasks and ge…

You are probably right with your example for a small project. The one I referred was a medium sized safety critical piece with dev/test staff of hundreds. My example was for such larger organization.

Small teams almost always worked out well for me if a single leader person was present to sort out initial problems.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#190

Post by ckastner » Wed, Oct 19, 2016, 7:12 AM UTC

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

> The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.)

The top-most comment on that page emphasizes this point:

Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of TDD (and, in fact, unit testing) are negligible. However, there's a non-linear benefit as systems become larger, particularly in the diagnosis of large and complex system failures.