Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An External Replication on the Effects of Test-driven Development [pdf]

people.brunel.ac.uk

Post reply on HN
Print view View original

161–170 of 332 posts

Re: An External Replication on the Effects of Test-driven Development [pdf]

#161

Post by EdiX »

hackuser wrote: post #98
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

> In a graduate-level engineering class, the students were making similar statements about all the studies we read

Yeah, that's because they are all terrible.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#162

Post by johan_larson »

The questions worth asking about techniques like TDD are "What problems does it fix?" and "What problems does it introduce?"

I would expect a determined attempt at TDD to solve the "no tests" problem, because it is so utterly insistent on tests. It should also solve the "don't know how to start" problem, because it de-emphasizes planning and design in favor of just jumping in; you write the tests, and then you do the bare minimum to make them pass.

That said, I would expect a TDD-based project to have the "bad architecture" problem: messy interfaces and sort of ad-hoc separation of concerns, because it makes no time for up-front analysis and design. It's always focused on the current feature and doing whatever it takes to make it work now.

In fairness, it does include a refactoring step, which is supposed to clean up the mess after the fact. Color me skeptical. Refactoring is hard, and people tend to do it on a large scale only when they have to.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#163

Post by NumberCruncher »

hackuser wrote: post #98
jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work;…

Scientific methods were invented to minimise the flaws in scientific work. If you ignore these methods and base your study on biased data you better forget calling your results scientific. Fortunately there are folks on HN reminding you on this.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#164

Post by EdSharkey »

mehwoot wrote: post #150
EdSharkey wrote: post #100

The * fire your QA team * dev team is the level 2 production support, and * get to continuous integration nirvana management fads have been sweeping through my Scrum enterprise for the last 18 months. Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedul…

The study is about the effects of "test first" vs "test later", not "writing tests" vs "not having tests".

Yeah, I tried to avoid specifying a methodology. I just couldn't resist throwing in a 'TDD' because it turns better quality results for me personally than 'test later'. My point in commenting was: testing comprehensively is not optional at my workplace due to the way the devs have suddenly been held so accountable.

Personally, I don't care how you test. Just get decent coverage over all the control paths and have a test suite that catches regression bugs. Ideally also have integration tests to prove your connections to your partners are good and maybe a smoke test to prove your code is wired up right once it hits an environment.

I'm still on the journey myself, so I can't claim to be an expert. I know there are people in my org that are suffering because of code quality issues. I call them incompetent because they're producing shoddy work products with lots of defects. A smart way to combat bugs and kill them for good is to test, so that's why I connect not testing to incompetence. A dumb way to combat bugs is heroics and working 18 hour work days trying to debug with "GOT HERE" logging.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#165

Post by yitchelle »

barbs wrote: post #7
geerlingguy wrote: post #5

Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc. But I know some people are wired differently; they'll think a lot mo…

> if it works for you and your coding style, awesome. But don't force it down my throat or act like it's the One True Path to clean code. This applies to so many things in Software Development - text-editors, variable names, plugins, architectures, operating systems...

Well, things start to get complicated and messy when the team size starts to grow and the technical debt starts to mount. Thats when it needs to goes from hacking to engineering.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#167

Post by kriro »

jdlshore wrote: post #71

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two year…

Sample size is 21 and they used Kruskal–Wallis (which is parameter free). Might not be the best tests (iirc. it's usually recommended for the exact combination of ordinal+nominal but I'd have to check) but it's suitable enough.

Also I don't think convenience sampling is bad in this case (and they clearly indicate they did it etc.) as it allows you to control for TDD experience and the like. I'd even go further and say the typical "lol students as participants" isn't an issue either.

"How effective is test-Driven Development. Making Software: What Really Works, and Why We Believe It" by Turhan et al. is a good meta analysis if you want more studies.

As an aside I like the fact that this is a replication study and that there have been a couple of these.

I mean worst case you can uses theses studies to guestimate better priors for your own Bayesian analysis :)

Re: An External Replication on the Effects of Test-driven Development [pdf]

#168

Post by hvidgaard »

kodfodrasz wrote: post #135
hirsin wrote: post #107

Earlier quoted context omitted.

Potentially contentious opinion, but one that's been echoed through our halls - Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled. It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably…

I have experienced the opposite. After the dedicated QA team was disbanded and repurposed to development, where every developer had to write tests for someone else's code, the code and designs started to become more testable, the tests eventually became simpler to maintain and understand. (the project was and embedded system) The "We" and "Them" distinctions made people in different teams and different roles ignore t…

There is no silver bullet here. For some teams it makes sense to integrate development and testing. For some teams it makes sense to have dedicated QA people. For some teams a different constellation is optimal. I know developers that are brilliant at the big picture, finishing the implementation however was lacking. I also know developers that cannot get the major architecture right, but they can finish tasks and get it shipped. pick one of each and a devops guy that can write tests, and you have a team of 3 that produce top quality software.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#169

Post by iUsedToCode »

The research seems low quality. Whenever i try creating something more complex than just a CRUD webapp, i'm always relieved after getting a significant code coverage.

It may be because i'm a medicore programmer (i mostly do hobby projects), but getting assurance that my 'small change here' didn't mess up anything major in a distant part of the system is quite relaxing.

Obviously i only test logic and usually write the tests after coding. It still helps with my flow.

Re: An External Replication on the Effects of Test-driven Development [pdf]

#170

Post by smallnamespace »

vacri wrote: post #159
smallnamespace wrote: post #115

Earlier quoted context omitted.

The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second lo…

This may sound harsh, but taking the researcher's difficulties into account is not our responsibility.

The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.

Post reply on HN