Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

251–260 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#251

Earlier quoted context omitted.

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

Absolutely, this. TDD is hugely valuable especially when new people join the project who don't understand and the best way they contribute is by adding tests!

That sounds like TLD, the opposite of TDD.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#252

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

When I first learned about TDD I thought the same (order of test and implementation does not matter that much). Yet when you do the TDD steps

* write a failing test (RED) * implement minimal functionality so all tests pass (GREEN) * REFACTOR

and overcome the initial revulsion you will get a ton of benefits that you didn't know existed. I'll first give a few that are more on the code quality side

* You have a complete test suite, without redundant tests * Tests are usually not that mock-focused [] as the implementation-after-test didn't introduce random roadblockers for testing. tests do assert the relevant properties (tests-later code bases often have code bases that don't assert much or properties you don't care about). * When you write your test first, you basically write a small example how it feels like to use the API you have in mind for solving your problem. This drastically improves quality.

Some non-technical benefits of TDD

* I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing. * TDD is programming gamification. You are rewarded with regular, small successes of having added another test and made it green. So whenever you ask yourself: OMG, have I made any progress in the last 3 days, you can actually tell: Yes, I added 15 tests and made them pass - I objectively got closer to implementing that feature. * Not an issue where I work, but have often heard about this from other places: If tests are there first, no one can tell you to not spend so much time on writing tests. TDD-style is protecting your boss from shipping that car without having the brakes checked. * The same line: TDD gives explicit room for refactoring.

[*] obviously one cannot avoid mocking especially when dealing with resources external to your code base, yet I have hardly seen TDD-originated tests that mocked

Re: An External Replication on the Effects of Test-driven Development [pdf]

#253

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

>or any style of automated tests, really, you don't really have to be driven by them

Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in just couple of screens) and because of that TDD must be compared not to other test methods, but to other methods of design.

Of course, that comparison will not show the supremacy of TDD for many reasons, like, for example, insufficient expressiveness of testing framework or domain specifics. TDD is useful as one of the many tools we have today, but it shall never be the only tool.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#254

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

TDD vs. test-after-code is a small distinction. 80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions. So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).…

> 80% of software development is designing correct abstractions/interfaces/APIs

Not sure if that claim is wrong or not, but I will state it anyway. If you write in TDD manner and you decide to change the API/abstraction a little bit because of something you didn't think of, you will have to correct both tests and the code. When you do test-after-code it's much easier (IMO) to hit correct abstractions and interfaces because you are working on a living thing, not some artificial test cases you have to figure out. Then you write tests to ensure those abstractions are respected and in working order.

TDD people, please correct me if I'm wrong.

EDIT: disclaimer: I'm a front-end developer working mostly in React and Redux

Re: An External Replication on the Effects of Test-driven Development [pdf]

#255
post #229

Earlier quoted context omitted.

I have a somewhat different experience. I've found TDD is useful on even tiny projects because using TDD forces you to write better code. It's really hard to reliably test things that use "magic" and side effects that affect parts of your code they shouldn't really be affecting, and really easy to test things that have well-defined and properly documented interfaces that only do one thing. Consequently writing tests,…

> Consequently writing tests, even if you never actually run them, makes your software better. It could be argued that an experienced programmer doesn't need TDD to write the same code that TDD could have produced.

That's true. Neither from system design, nor from quality perspective TDD is necessary to produce good code. However, it may still add some value even for experienced programmers, when it's easier to write a test than to build the pure mental model of the domain or the algorithm.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#256
post #61

Earlier quoted context omitted.

Exactly my experience - I've done TDD for 3 different types of calculation functions in 3 different languages over the years, and all were beneficial: financial risk calculations in C++, report data calculations in PHP and billing calculations in JavaScript. On the other hand, TDD for operations that involve I/O (including user interactions) were not helpful at all.

That would probably be better handled by functional/e2e testing?

TDD doesn't assume you are using unit tests.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#257

Earlier quoted context omitted.

>You can have tests without TDD ? I've worked on several projects without this and the tests done without TDD tend to be of higher quality. I noticed a common anti-pattern of "write the code, run the code, copy the output of the code, paste it into a test and write an assert to check that the output was precisely what came out". This was brittle, it killed the self-documenting aspect of the tests and it often conceal…

>I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in. How do you know you made API mistakes from writing unit tests ? API mistakes become apparent when you integrate stuff and use the API in conjunction with other things (using API in isolated scenarios like unit tests is not really insightful you can figure out those fl…

I usually don't do unit test driven development actually. I typically do integration test driven development where the integration test sets up an environment where the code is either interacting with the real thing or a realistic mock version.

I don't find unit tests all that useful for integration code - either as tests that find bugs or for doing TDD.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#258
TDD means 'management' can't drop the tests being written due to 'timescales'. If they're done up-front, they will be there.

It's also one reason that TDD isn't done, because given a few weeks to complete an impossible deadline means that tests are the first ideal to be dropped.

It's not the correct way to do things, but all of these studied tend to ignore the 'real world'.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#259
post #171

Earlier quoted context omitted.

I have a somewhat different experience. I've found TDD is useful on even tiny projects because using TDD forces you to write better code. It's really hard to reliably test things that use "magic" and side effects that affect parts of your code they shouldn't really be affecting, and really easy to test things that have well-defined and properly documented interfaces that only do one thing. Consequently writing tests,…

I would humbly suggest that because you and parent (and other people here) have such a different positive experience with TDD, that you cannot quite agree when it works, only that it sometimes works, then this is a big indication that TDD is just a placebo. But maybe that's what's needed - a placebo to feel you better as a programmer, and thus making you more productive through your feelings.

I can't speak for the other person, but from my reading, we don't actually have very different experiences. We seem to agree on the value of testing of large/complex project, but have a (minor) difference of opinion over the value of testing in very small projects. Indeed, I'm pretty sure we're deep into hair-splitting territory, as I do agree that TDD'ing small projects can lead to better code, I just don't think the pay off is as obviously big as it is for more complex ones.

It's like observing two fans debate which Star Wars movie is the best and the merits of Jar Jar Binks, and concluding that because they disagree, in fact all Star Wars movies must be pretty bad.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#260
The use of students in SE research is a hot topic, see, for example Fietelson's review: https://arxiv.org/abs/1512.08409.

Practitioners have a problem recruiting subjects. There is often a tradeoff between applying more rigorous experiment design and using convenience sampling (students) versus sacrificing controlled environments (so that professionals would actually join the study).

It's easy to condemn work like this but there's no other option. In this case the researchers chose to replicate a study (which often risks similar ire for telling us nothing new) with a commendable level of rigour and have provided more evidence that, for the scope of experiments we can construct that TDD is probably no different to TLD when using a population of relatively unqualified developers (students).

As to the problem being trivial, what else can be done? There's a finite time you can ethically expect participants to give to you, even if you pay them. If anything the criticism of this work is better directed at the limitations academics are forced to bear.

Post reply on HN