Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

291–300 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#291

Post by noxToken »

hirzel wrote: post #279
awjr wrote: post #271

Earlier quoted context omitted.

This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.

Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?

Yes. Some people do not know how to write extensible software. They solve the issue at hand without any regard to big picture, so when something needs to be added (like test) the code needs to be heavily refactored to accommodate the change. As others have stated, this usually(!) isn't a big deal when the project is small with a single contributor. However, writing the tests last without any forethought to what the tests are or how they might be implemented will incur debt when the code needs to be refactored.

Conversely, extensible code can have the tests written before or after without much change to the amount of technical debt. The code is written to adapt to change, so tacking on robust tests wouldn't require too much technical debt.

Note that there is a balance. If you know your goals and have no reasonable expectation for them to change, then making everything extensible is unnecessary overhead. You can also have a mix of abstract functions that are extensible alongside immutable functions that handle something specific. It's all about what works best for your project.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#292

Post by [deleted] »

hirzel wrote: post #279
awjr wrote: post #271

Earlier quoted context omitted.

This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.

Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?

[deleted]

Re: An External Replication on the Effects of Test-driven Development [pdf]

#293

Post by hackuser »

adwn wrote: post #203
hackuser wrote: post #153

Earlier quoted context omitted.

While I can see your point of view, perhaps that is an ideal and not a realistic view of how science actually works. Similarly, someone might have an ideal view of software development, but if they saw how it actually worked and read actual in-production code ... and I think that describes every profession. Science does work pretty well; it predicts things with accuracy and reliability otherwise unknown by humanity (…

You're conflating the "hard" sciences (which rely on reproducible experiments to explore natural laws) with the "soft" sciences (which rely on studies to explore human behavior). The track record of the latter is significantly worse than that of the former. Software engineering, and by extension this study, falls firmly into the soft sciences territory. You should be a lot more sceptical of supposed facts in soft sci…

I know the differences you are referring to talking about. I mean all sciences, but really I mean that we need to hear from actual practitioners about how they really work.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#294

Post by crdoconnor »

collyw wrote: post #277
crdoconnor wrote: post #228

Earlier quoted context omitted.

I have the same positive experience as both of the above commenters. I've found that there are many other variables which can change its effectiveness though: * The type of code I'm working on * How well I remember/understand the full requirements. * How much of the testing framework is built already and how much I have to build. * The type of test I'm writing. * How new the project is.

Yes, TDD seems to involve having a decent idea of the requirements (very un-agile). Many projects are making it up as they go along and iterating to a reasonably correct outcome.

It means having a clear idea about the requirement you're implementing at that point in time (not un-agile). It doesn't mean that you have to create a big set of requirements up front (un-agile). You can just pick a story, convert it into a test and implement the code, rinse and repeat.

Writing the test also gives you a second chance to think deeply about the requirement and fix any problems with it before wasting time implementing the wrong thing.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#295

Post by mikegerwitz »

mseebach wrote: post #108
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

Tests provide certain guarantees where compiler proofs (e.g. compile-time errors) are absent. You don't need a large project to realize those guarantees.

Tests aren't just for initial correctness---their benefit is largely in maintenance. They allow you to make changes or refactor the system with guarantees that the covered portions still operate as they were originally designed. There is really no excuse for those types of breaks.

Then there's the team setting: I'm not the only one writing my code. Someone else has to get in there and make changes. I might have to come back months or years later and make changes. At work, we have five developers touching over 100 distinct projects at any point. We have observed that the lack of tests essentially guarantees breaks---we have QA, and there is a direct correlation between bugs and whether code is comprehensively tested.

With regards to TDD: we also observe time and time again that writing code after tests yields untested behavior. There are always odd cases that might not be immediately obvious. There might be a bug that was inadvertently introduced, and now that bug is tested as part of the implementation. Certain branches may not be fully tested. I've seen comments here about this being an experience thing---that more senior developers won't have this problem. That's essentially saying that one is infallible; it doesn't make sense.

TDD also prohibits rushing. Not writing tests is also an excuse to rush the implementation and produce a shitton of unnecessary code. Tests slow you down and force you to think.

Small projects grow. If you don't write tests upfront, when do you write them? We develop incrementally---all of our projects start small, and some of them will remain small for perhaps a year or more until we revisit it. Writing tests at that point loses a lot: we may have forgotten the details of the implementation by then, and the person who originally wrote it might not be involved at all in the changes.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#297

Post by z3t4 »

err4nt wrote: post #51
tspike wrote: post #14

My experience has been that TDD is worthwhile when working with notoriously slippery whack-a-mole functions like handling time or money. The time saved by catching regressions vastly outweighs the time taken to implement the tests. In contrast, TDD has been a waste of time for me for UI-based work, as the effort needed to properly expose the functionality under test is too great and the requirements and design change…

Can you elaborate more? Im a frontend guy and sometimes I write JS plugins for the browser (that run after page load, and apply based on how things render on the page, or bawed on user interaction with the page) and non-frontend folks tell me I need tests for my code, or dont want to look at it until I have test. How would I build a tewt for this, other than a functional test that runs in-browser and the test is whet…

First off, if your code don't have bugs, then that is preferable! In my experience with front web dev the issue has always been different (old) devices. Testing could be done by browser macros that simulate input, like mouse movement, click on x, y, z, etc. Then take screen-shots to see if something is off. You can also make your widgets throw errors (1) and call home via XMLHttpRequest if an error is detected.

1) http://www.webtigerteam.com/johan/en/blog/error_driven_devel...

Re: An External Replication on the Effects of Test-driven Development [pdf]

#298

Post by awjr »

hirzel wrote: post #279
awjr wrote: post #271

Earlier quoted context omitted.

This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.

Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?

Writing well designed functions that do one thing enables good testing.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#299

Post by Semiapies »

simula67 wrote: post #133
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.

Since when is pointing out the serious flaws of a study "rhetoric"?

I'm ambivalent on TDD, but the arguments defending this study here have been ridiculous and anti-intellectual.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#300

Post by parfe »

apatters wrote: post #127
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

>Why do comments like this get upvoted so much?

They're dressed up versions of "correlation != causation". Low thought comments appealing to people with not-even stats 101 knowledge.

There is no such thing as a perfect study, and a generic version of the OPs comment could be copy and pasted on any study ever. Unless you study the entire population of the planet, you're going to miss subgroups. Unless you study the entire population of the planet, you're going to need some sort of selection criteria. Unless you have infinite funding and time, you're going to need to make trade offs and sacrifices in your experiment design.

A study will disclose these shortcomings for readers to balance the significance of results against.

Take this complaint from OP:

>* The sample size was tiny. (20 students)

What sample size would satisfy him? Why is 20 too small? 40? 80? 1037? Is he basing his opinion of a proper sample size on his gut? 20 just doesn't feel right?

Post reply on HN