Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

271–280 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#271

Post by awjr »

ckastner wrote: post #190
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

> The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.) The top-most comment on that page emphasizes this point: Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of…

This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible.

However, the moment it becomes more than just a one-off project this debt becomes a problem.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#272

Post by Beltiras »

StreamBright wrote: post #226
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

I agree with you on this. The study is irrelevant from the statistical point of view. However, there are other considerations with TDD, probably on a more theoretical level that: * There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?) * There is one thing common for sure with all of the bugs you catch in production, they a…

If you could bang a car against a guard rail without incurring the cost of a car, you would.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#273

Post by StreamBright »

regularfry wrote: post #240
StreamBright wrote: post #226

Earlier quoted context omitted.

I agree with you on this. The study is irrelevant from the statistical point of view. However, there are other considerations with TDD, probably on a more theoretical level that: * There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?) * There is one thing common for sure with all of the bugs you catch in production, they a…

To your first point, the usual response is that there's no other major engineering branch which demands quite such extensive changes be possible after you've broken ground. To extend your analogy a little, nobody takes a car production line and expects the people running it to be able to switch to helicopters without starting over.

I agree, and it does not happen to my projects either. We agree in advance on the set of features for a certain version and we deliver that. If you doing much car building when you actually need a helicopter than there is something wrong.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#274

Post by vacri »

smallnamespace wrote: post #170
vacri wrote: post #159

Earlier quoted context omitted.

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second lo…

This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.

Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#275

Post by vacri »

icebraining wrote: post #173
vacri wrote: post #159

Earlier quoted context omitted.

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second lo…

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? Hackatons seem to manage. Why not set one up?

Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#276

Post by vacri »

vidarh wrote: post #223
vacri wrote: post #159

Earlier quoted context omitted.

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second lo…

You could start by finding a corporate sponsor with a lot of developers and a vested interest in finding out which methods will be best for them. The advantage of that being that you could even test it on a real project (the sponsor would "just" need to be willing to dedicate twice the number of developers to a suitably small project). It'd still not be easy, and of course there'd still be issues (e.g. is there anyth…

Corporate places do research like that all the time; they just don't publish them all that often. A corporation with a vested interest is a corporation with a competitive interest.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#277

Post by collyw »

crdoconnor wrote: post #228
js8 wrote: post #171

Earlier quoted context omitted.

I would humbly suggest that because you and parent (and other people here) have such a different positive experience with TDD, that you cannot quite agree when it works, only that it sometimes works, then this is a big indication that TDD is just a placebo. But maybe that's what's needed - a placebo to feel you better as a programmer, and thus making you more productive through your feelings.

I have the same positive experience as both of the above commenters. I've found that there are many other variables which can change its effectiveness though: * The type of code I'm working on * How well I remember/understand the full requirements. * How much of the testing framework is built already and how much I have to build. * The type of test I'm writing. * How new the project is.

Yes, TDD seems to involve having a decent idea of the requirements (very un-agile). Many projects are making it up as they go along and iterating to a reasonably correct outcome.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#278

Post by spatulon »

ivan_gammel wrote: post #253
mseebach wrote: post #108

Earlier quoted context omitted.

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

>or any style of automated tests, really, you don't really have to be driven by them Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in j…

This benefit of test-driven design is not obvious (at least it wasn't for me) until you really experience it. Several years back I wasn't aware of the concept of dependency injection, but independently discovered it while trying to unit test some error logging functionality. Previously I'd always used a global/singleton logger object. That always felt wrong, but I always assumed it was necessary until writing unit tests forced me to avoid that design.

Also note that TDD is not the only way of getting this benefit. IMHO Paul Graham's essay Programming Bottom Up[1] and Casey Muratori's concept of 'compression-oriented programming'[2] both espouse this same idea of improving API design by "writing examples of their use."

[1] http://www.paulgraham.com/progbot.html

[2] https://mollyrocket.com/casey/stream_0019.html

Re: An External Replication on the Effects of Test-driven Development [pdf]

#279

Post by hirzel »

awjr wrote: post #271
ckastner wrote: post #190

Earlier quoted context omitted.

> The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.) The top-most comment on that page emphasizes this point: Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of…

This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.

Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process?

In terms of technical debt, does it matter when the tests are written?

Re: An External Replication on the Effects of Test-driven Development [pdf]

#280

Post by user5994461 »

embwbam wrote: post #88
paulddraper wrote: post #85

Earlier quoted context omitted.

TDD vs. test-after-code is a small distinction. 80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions. So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).…

This is why designing the interface in a language with a powerful type system provides the same quoted benefit of TDD. It helps you think about the interface before you move to implementation. In my experience, using a type system to do this requires much less effort and refactoring.

> This is why designing the interface in a language with a powerful type system provides the same quoted benefit of TDD.

Yes. TDD came from developers who worked in weak/dynamic typed languages.

It is mandatory to discover errors, that are otherwise discovered instantly for free by compilers in the stronger typed languages.

Strangely, the TDD folks never outline this point.

Post reply on HN