Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

71–80 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#71

Post by jdlshore »

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN.

* The sample size was tiny. (20 students)

* The participants were selected by convenience. (They were students in the researcher's class.)

* The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two years' experience.)

* The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.)

Maybe, maybe you could use this to draw conclusions about how TDD affects novices working on simple algorithmic problems. Given the tiny sample size and sampling by convenience, I'm not sure you can even draw that much of a conclusion.

But it won't tell you anything about whether or not TDD impacts development time or code quality in real-world development.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#72

Post by qaq »

donw wrote: post #68

Am I correct in reading that they performed this experiment only for two days, and entirely with graduate students? If so, they have missed the point of TDD. In the short term, TDD probably doesn't make a difference, one way or another. But software as a business is not a short-term game. I would love to see a study where the participants are, over a period of six months, given the same series of features (including…

Phoenix (Chris is in TLD camp and Phoenix has very good test coverage)

Re: An External Replication on the Effects of Test-driven Development [pdf]

#75

Post by Tyr42 »

greesil wrote: post #21

This is the kind of stuff that in the aggregate you can't show a relationship, but I bet if you controlled for type of project one would see some interesting results. Anecdotally, I know some firmware engineers that shit out the buggiest code I have ever seen, and test driven development would have definitely improved the customer experience. Because when the engineers have literally no tests other than trying stuff…

I just came out of an embedded system-sy project, and I did have some tests for my ringbuffs, and sprintf.

But tests for any of the interactions between subsystems is quite problematic. And then testing on the device might be problematic due to space constraints, but testing on a simulation is also problematic... I don't know how you'd realistically test it.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#76

Post by madeofpalk »

bunderbunder wrote: post #25
geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

For me it's a highly situation-dependent thing. Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth betw…

Yes.

When doing UI code (with something like React) it's next to impossible to do TDD - you can't know the structure until you actually implement it.

If I'm fixing a bug in some sort of non-ui code, I often find TDD to be very helpful. Create a test case that demonstrates that failure, then fix the implementation to make the test pass.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#77

Post by olalonde »

bunderbunder wrote: post #25
geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

For me it's a highly situation-dependent thing. Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth betw…

> Sometimes writing tests first helps me think about the high-level design of my code.

I'm personally a fan of "readme driven development" (https://news.ycombinator.com/item?id=1627246). It's a nice compromise between tons of upfront planning and nothing at all.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#78

Post by wagonman »

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

Thank you for looking into the details.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#79

Post by [deleted] »

dtwhitney wrote: post #50
barbs wrote: post #7

Earlier quoted context omitted.

> if it works for you and your coding style, awesome. But don't force it down my throat or act like it's the One True Path to clean code. This applies to so many things in Software Development - text-editors, variable names, plugins, architectures, operating systems...

nope - vim

[deleted]

Re: An External Replication on the Effects of Test-driven Development [pdf]

#80

Post by i__believe »

Alex3917 wrote: post #67
geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

> I think the chasm exists between _untested_ code and code that has tests. IMHO untested code isn't always a bad thing. Anything related to privacy, security, or data integrity should be heavily tested. But beyond that, code should need to earn its tests. (At least in a web startup context.) After all, the core of agility is being able and willing to go in a different direction when something isn't working.

I think there are a lot of nuance to this statement. After a few years on the start up scene, I believe that if you are using a "duck" typed language your core concepts should be tested without exception (for example rails and django models). The farther you get away from these core concepts, the less your tests need to be there and the more likely that code is to change.

In addition, if you are truly in the earliest stages of your startup (pre-revenue/traction) the tests are not needed -- but you have to be _very_ aware that you are making a tradeoff for sheer velocity and you will have to pay the price later on. I've seen too many startups fail to pay the price and it comes back to haunt them and reduces their overall velocity.

Post reply on HN