Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

151–160 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#151

Post by Strilanc »

hackuser wrote: post #98
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

> the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc.

That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect.

I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#152

Post by zorked »

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

Criticism to the study goes both ways. It COULD be that with a larger sample size, more meaningful project and more experienced developers the result would be "TDD actively hampers productivity by 70% and leads to more bugs compared to test-last".

Being mindful of the conditions of the study is important. Shrugging it off isn't helpful in the least.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#153

Post by hackuser »

jdlshore wrote: post #137
tlarkworthy wrote: post #131

Earlier quoted context omitted.

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…

While I can see your point of view, perhaps that is an ideal and not a realistic view of how science actually works. Similarly, someone might have an ideal view of software development, but if they saw how it actually worked and read actual in-production code ... and I think that describes every profession.

Science does work pretty well; it predicts things with accuracy and reliability otherwise unknown by humanity (AFAIK). It's an interesting question: If not by hewing close to this ideal, how does it actually achieve results?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#155

Post by sanderjd »

apatters wrote: post #127
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

Perhaps oddly, it turns out that presenting no data is better than presenting data with poor foundation. Comments like this get upvoted because many of us appreciate someone pointing out when there's no "there" there before we've spent our own time reading.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#156

Post by beambot »

jdlshore wrote: post #137
tlarkworthy wrote: post #131

Earlier quoted context omitted.

It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing ever…

If studies don't need to be rigorous, what's the point of doing them? Why not just write a persuasive essay instead? Is the format of a "study" just a rhetorical device, a glammed-up appeal to authority? The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anyt…

> The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not.

That's where you're wrong... even with "careful statistics", the point of sample-based studies is to disprove the null hypothesis with some confidence level. There is no requirement that it be 100% confident. In fact, with p(See, it's easy to get this wrong!)

Re: An External Replication on the Effects of Test-driven Development [pdf]

#157

Post by algesten »

apatters wrote: post #127
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

uhm. lol? are you saying a study can't be criticised by anything less than a counter study?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#158

Post by snovv_crash »

Alex3917 wrote: post #67
geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

> I think the chasm exists between _untested_ code and code that has tests. IMHO untested code isn't always a bad thing. Anything related to privacy, security, or data integrity should be heavily tested. But beyond that, code should need to earn its tests. (At least in a web startup context.) After all, the core of agility is being able and willing to go in a different direction when something isn't working.

If your code doesn't have tests then you can't refactor because you don't know if you broke something. Getting the right level of tests is important, they should check the interface and not the implementation, otherwise I agree, you can't refactor anything. But if you test your interfaces the same way that real code would use it then it actually makes you more agile, because it gives you the peace of mind to do major changes.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#159

Post by vacri »

smallnamespace wrote: post #115
Ar-Curunir wrote: post #101

Earlier quoted context omitted.

Exactly this; I find too many HN comments to be critical in a non-constructive manner.

The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done.

So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group?

Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#160

Post by Confusion »

The comments to that story are pretty good.

An interesting question is: why does TDD fail in such experiments (it does so unexpectedly consistently), even when many developers feel it has benefits when they practice it?

There is no silver bullet, so there must be circumstances in which TDD does not work. And conversely, the central question is: under what circumstances does TDD work? What are the preconditions?

Post reply on HN