An External Replication on the Effects of Test-driven Development [pdf] • Hacker News Board

Re: An External Replication on the Effects of Test-driven Development [pdf]

#101

Post by Ar-Curunir » Wed, Oct 19, 2016, 4:48 AM UTC

hackuser wrote: post #98

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

Exactly this; I find too many HN comments to be critical in a non-constructive manner.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#102

Post by unclebucknasty » Wed, Oct 19, 2016, 4:48 AM UTC

hackuser wrote: post #98

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

Indeed. It is predictable and--as a discussion point--stifling.

It's far more interesting to discuss the actual merits than to dismiss the subject out of hand.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#103

Post by Rapzid » Wed, Oct 19, 2016, 4:52 AM UTC

My default is to not write many tests at all during the experimental, build-out phase. I'm not looking for exact or bug-free software, I'm trying out different API's, aggregates, and architecture in general. Needing to refactor tests every time I want to make a drastic change is... Well, you know. AS somebody else pointed out, this architectural stuff is probably actually much harder to nail down than just writing code that works. This is not limited to very initial build out but could apply to big refactors as well.

After and during the experimental phase, it depends. Both before and after I may write tests before or "test with" for gnarly logic or algorithm-y stuff. Otherwise and in addition I do copious amounts of manual testing. Manual testing is a must for much of what I do, so I augment or substitute automated testing as appropriate. Automated testing is great, but sometimes the overhead is too expensive.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#104

Post by sayrer » Wed, Oct 19, 2016, 4:53 AM UTC

In "Realizing quality improvement through test driven development: results and experiences of four industrial teams", an MSR researcher found that TDD did reduce defects in his study, but also came at a large cost in time-to-ship.

https://www.microsoft.com/en-us/research/exploding-software-...

This finding contradicts the headline. TDD impacted both development time and code quality in that study.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#105

Post by dcosson » Wed, Oct 19, 2016, 4:53 AM UTC

bunderbunder wrote: post #25

geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

For me it's a highly situation-dependent thing. Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth betw…

I find the same thing. It's almost like writing an essay, I need to sketch out an outline first, which is often getting just far enough to convince myself that the approach is reasonable and there aren't any surprises ahead. Once it's time to crank out the details of individual sections, the work often starts to feel monotonous and that's where TDD can be invaluable for helping me to stay focused and comprehensively work through all the edge cases.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#106

Post by ericdykstra » Wed, Oct 19, 2016, 4:54 AM UTC

hackuser wrote: post #98

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc.

This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise."

I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#107

Post by hirsin » Wed, Oct 19, 2016, 4:55 AM UTC

EdSharkey wrote: post #100

The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…

Potentially contentious opinion, but one that's been echoed through our halls -

Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled.

It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably outperform (in code output and reliability) 10 devs spending 70% of their time on testing.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#108

Post by mseebach » Wed, Oct 19, 2016, 4:56 AM UTC

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#109

Post by flukus » Wed, Oct 19, 2016, 4:56 AM UTC

candiodari wrote: post #60

brightball wrote: post #19

Earlier quoted context omitted.

Agree here. Tests the stuff that needs testing.

That means you can't use coverage tools, so this approach makes the tests unquantifiable. It is not how most major dev shops work, because of the bean counters and the "senior" developers/leads/managers/... who seem to want nothing but appeasing beancounters. It also produces superior software in my experience.

> That means you can't use coverage tools, so this approach makes the tests unquantifiable.

So it's win win then ;)

Re: An External Replication on the Effects of Test-driven Development [pdf]

#110

Post by jcoffland » Wed, Oct 19, 2016, 4:58 AM UTC

shade23 wrote: post #92

I have spent almost 3 years now writing code(with very few or no tests) and my current organization stresses on agile practices a lot.I encountered TDD from here.So I would like to chip in here too. TDD solved a major problem for me which I have seen a lot of people suffer with. _Where do I start ?_ . The thing is TDD and refactoring go hand in hand. I cannot imagine doing TDD if I was not using an IDE like Intellij…

> When you normally start writing code first(typical TLD) then you need to have a plan before hand. This plan cannot change much because you really do not get feedback till you complete major segments of the code.

A good plan is modular and therefore flexible. I get lots of feedback as I'm writing the code. When I find myself writing very similar code over and over or an API feels unwieldy I take that feedback and refactor without delay.

I think what really happens in big groups of developers is that TDD forces everyone to delay agreement on the interfaces between code modules until the process of writing tests has uncovered most of the problems. With out TDD the devs who just want to get their part done and go home plow ahead with the first draft of the API locking it in stone before it's been vetted and then dig in their heels creating technical debt. TDD appears to be the saviour.