An External Replication on the Effects of Test-driven Development [pdf] • Hacker News Board

Re: An External Replication on the Effects of Test-driven Development [pdf]

#301

Post by falcolas » Wed, Oct 19, 2016, 2:03 PM UTC

js8 wrote: post #171

onion2k wrote: post #126

Earlier quoted context omitted.

I have a somewhat different experience. I've found TDD is useful on even tiny projects because using TDD forces you to write better code. It's really hard to reliably test things that use "magic" and side effects that affect parts of your code they shouldn't really be affecting, and really easy to test things that have well-defined and properly documented interfaces that only do one thing. Consequently writing tests,…

I would humbly suggest that because you and parent (and other people here) have such a different positive experience with TDD, that you cannot quite agree when it works, only that it sometimes works, then this is a big indication that TDD is just a placebo. But maybe that's what's needed - a placebo to feel you better as a programmer, and thus making you more productive through your feelings.

> TDD is just a placebo

This is my experience. I've watched teams do TDD, only to come up against their first refactor and have to change 80% or more of the tests. Just the other day, they were griping about a code review that they had to do which involved changes to 2,000 lines of test code over a code change with minimal functionality change.

TDD has the effect of making you think differently about how you code. Thinking about how you write code is a good thing, but you don't need TDD to make you think differently about your code. You just need to think.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#302

Post by gostonethecrows » Wed, Oct 19, 2016, 2:12 PM UTC

There are many problems with this study, but for me the most glaring is the definition of quality that they measured. It was purely whether the program performed as expected. This is obviously an important part of code quality, but not the only one. Most proponents of TDD say that its greatest benefit is creating clean, easily maintained code. So this study didn't even attempt to test the benefit that TDD claims to provide.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#303

Post by roadman » Wed, Oct 19, 2016, 2:21 PM UTC

I was eager to read this paper but found little substance in it.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#304

Post by wpietri » Wed, Oct 19, 2016, 2:22 PM UTC

zzalpha wrote: post #54

wpietri wrote: post #47

Earlier quoted context omitted.

> If you write testable software and actually write the tests, the end result is the same whether you test first or test later. This is definitely not my experience. As Kent Beck says, TDD is a design technique. It forces you to always start thinking of the code from the outside of the unit. If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implement…

If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implementation. With test last, I'm also less likely to test everything well; after the implementation is done, I believe it works. Will you? Are you sure? Do you have data to back your supposition? My contention would be that at the end of the day, the requirements of the interface to support unit te…

> Do you have data to back your supposition?

That's my experience. I started doing TDD more than 10 years ago, and it took me about a year to fully make the switch from test-after to test-first programming. I regularly try experiments with different personal code bases.

If you're arguing from personal experience, that your code ends up just the same either way, good for you. But I suspect you're arguing from theory here.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#305

Post by wpietri » Wed, Oct 19, 2016, 2:26 PM UTC

problems wrote: post #87

wpietri wrote: post #47

Earlier quoted context omitted.

> If you write testable software and actually write the tests, the end result is the same whether you test first or test later. This is definitely not my experience. As Kent Beck says, TDD is a design technique. It forces you to always start thinking of the code from the outside of the unit. If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implement…

Perhaps it depends on how you implement each. If your idea of test last is writing a full large module and then trying to test that all at once, you might wind up with a different result than if you wrote an individual function or two and tried to test only those at once. Getting the testing as close in time to the implementation as possible probably makes a bigger difference than first vs last.

I definitely agree with that. If there is somebody who actually writes a few units of production code, a few units of test code, and then refactors, then I expect things would be pretty close.

However, for me that's a hard set of behaviors to stick to. If I let myself say, "Oh, I'll test that later" I'm likely to get lost in the problem, write bunch of production code, and then say, "Well, it already works, but I guess I'll add some tests." There's no clear stopping point.

Test-first programming, on the other hand, is easier for me to stick to. If I don't have a broken test, I write one. If I have made the test pass, it's time to write another test before I write more production code.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#306

Post by omouse » Wed, Oct 19, 2016, 2:45 PM UTC

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

At the same time we should still praise them for trying to replicate a study, which doesn't happen nearly often enough in the computer science field (aside from algorithms and data structures studies).

Re: An External Replication on the Effects of Test-driven Development [pdf]

#307

Post by piaste » Wed, Oct 19, 2016, 2:48 PM UTC

crdoconnor wrote: post #237

rubber_duck wrote: post #188

Earlier quoted context omitted.

You can have tests without TDD ? TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests. Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the…

>You can have tests without TDD ? I've worked on several projects without this and the tests done without TDD tend to be of higher quality. I noticed a common anti-pattern of "write the code, run the code, copy the output of the code, paste it into a test and write an assert to check that the output was precisely what came out". This was brittle, it killed the self-documenting aspect of the tests and it often conceal…

> I've worked on several projects without this and the tests done without TDD tend to be of higher quality.

Do you mean "with TDD", by any chance?

The rest of your post says that you observed better results when tests were written before the code (rather than later). But as far as I understand, writing the tests before the code is indeed one of TDD's commandments.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#308

Post by omouse » Wed, Oct 19, 2016, 2:59 PM UTC

EdSharkey wrote: post #100

The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…

>As far as I'm concerned, in an environment like mine, where developers can no longer hide their incompetence behind bureaucracy like a QA team, it is official insanity to not spend inordinate amounts of development time writing automated tests. You should be spending 70% of your dev time writing tests and doing devops and 30% writing features.

Highly agreed, anytime there has been QA available, the code base starts to look more hacky and fewer tests are written and only the tests that cover the happy path are written. When things are looking bleak, that's when I start to think we should have certifications for developers, optional of course, like the PMP or CAPM, but they should be there as a rough indicator that a dev won't skimp on QAing their own code.

>I read in these comments a lot of bellyaching about how much time it takes to write tests. First, TDD is a skill that you can get good at, and it won't take as much time as you think once you get good. Second, I just don't think you have a choice to not test comprehensively when escape defects become a mark of shame in the organization.

After years of looking at tests written poorly, I've come to the conclusion that you have to treat test code like your regular code and do code reviews on it and refactor the heck out of it.

If you don't, you'll end up in the situation I'm in, where our company has almost 4000 spec tests in ruby/rails that can take an hour to run on a local dev box and still take 20 min when parallelized across 5 boxes. Shaving off a couple of seconds here and there by removing unneeded tests or refactoring to simplify them or to share mock objects is a worthwhile endeavour but because they're "just tests", the time to do that isn't allocated and everyone on the team merrily continues to write and modify tests in a sloppy way.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#309

Post by wirrbel » Wed, Oct 19, 2016, 3:00 PM UTC

nhaehnle wrote: post #268

wirrbel wrote: post #252

Earlier quoted context omitted.

When I first learned about TDD I thought the same (order of test and implementation does not matter that much). Yet when you do the TDD steps * write a failing test (RED) * implement minimal functionality so all tests pass (GREEN) * REFACTOR and overcome the initial revulsion you will get a ton of benefits that you didn't know existed. I'll first give a few that are more on the code quality side * You have a complete…

> I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing. This made me think. There are many axes of software development, and aside from TDD/non-TDD axis, there's also the bottom-up vs. top-down axis. By top-down, I mean you start with a skeleton that provides an interface to the outside world, and the fill in the implementation details. Bottom-up, on the other hand, means yo…

This reminds me of https://twitter.com/marick/status/787402452848873472

I dont think writing Tests first makes you work top down or bottom up.

If you want to work bottom-up (I like to work bottom up) you just start by writing tests for your bottom up functionality.

And of course I don't want to impose TDD on others, but I would like others to try it out sincerely (recommending kent Becks book for this. Don't do it without guidance, many people who reject TDD after trying it out were doing some kind of Test-first method, but didn't get the benefit because they were not aware of stuff like get one test at a time to complete, etc.)

---------

> shouldn't be taken as dogma.

Life in software engineering becomes much more relaxed when you realize that every piece of advice and technology is in fact based on dogmas and is not based on sound empirical, statistically relevant evidence :-) (apart from the pure math part).

It is just really really hard to study software quality in an objective way that satisfies scientific criteria. So we are kind of stuck with hearsay in IT

So TDD is kind of dogmatic. So is OOP, or the rejection of gotos.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#310

Post by Silhouette » Wed, Oct 19, 2016, 3:01 PM UTC

icebraining wrote: post #179

simula67 wrote: post #133

Earlier quoted context omitted.

One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.

Nope, there are also studies (done by Microsoft and IBM) indicating that TDD helps: https://www.infoq.com/news/2009/03/TDD-Improves-Quality

Careful: sometimes even widely cited studies like Nagappan don't actually show what they are misreported as showing either. For example, if you look at the development processes being compared in the cases examined, often they aren't quite TDD as widely described and advocated.