I don't know if TDD is effective. It seems effective on my single-person projects, less so on multi-person projects.
However, TDD is pretty effective in working with chatGPT. I always tell it to write the tests first.
What follows from empirical software research?
21–27 of 27 posts
Re: What follows from empirical software research?
#22"Empirical software research" could mean a bunch of different things. This article is about studying people writing software, not about software used for research in empirical sciences, and not about research into computer science. I'm confident the answer to what follows from that is "nothing yet" based on various conference talks. Studying developers (or in a worse case students) writing software doesn't seem to be…
I think this is what is being meant;
Wikipedia: https://en.wikipedia.org/wiki/Empirical_software_engineering (ESE)
Popular: https://www.americanscientist.org/article/empirical-software...
Microsoft has a ESE group with some interesting publications. I had read/downloaded a couple a while ago which unfortunately i can't now remember nor locate. But this is a good starting point - https://www.microsoft.com/en-us/research/publication/belief-...
Re: What follows from empirical software research?
#23One of the most important lessons I've learned in my career is that if a common problem has existed for a long time, a simple solution has probably been hiding in plain sight for a long time somewhere you haven't thought to look. I don't know anything about productivity research, but the fundamentals of defect density were figured out decades ago [1]. > (a) there’s no difference between test-first and test-after but…
Replying to my own comment to explain how it directly relates to the asked question - what to do with empirical research?
The key takeaway is to evaluate everything in terms of sensitivity because it gives you design insight. The above quoted study identifies one insensitive factor (order) and one sensitive factor (iteration length) of interleaving test and code. Now you can choose, modify, or design any test method as long as you do short interleaving. The order is something you don't have to worry about. If the research doesn't reveal anything about the sensitivity, don't worry about it until someone figures it out, your beliefs are probably wrong and inconsequential anyway.
Re: What follows from empirical software research?
#24Earlier quoted context omitted.
If you throw that in, you’d need to throw the TDD example out. A stronger test of your arguments would be, eg, research on productivity impact of generative AIs like copilot. (This would grow your example to match your argument, whereas my initial post would shrink your argument to match your example.)
The TDD example was just an example. I wasn’t taking the actual finding of the TDD research I was talking about a hypothetical finding. You can replace the examples with AIs like copilot and stipulate a large effect size. Nothing I said would really change. You still have to look at your desires to figure out what you ought to do given that research.
We're talking past each other, here.
Your article presents a sufficient argument for dismissing the TDD example. Tossing effect size in, your argument still applies for dismissing an ideal study.
My point is not that what you said didn't suffice, it's that the philosophically heavyweight arguments weren't necessary. They were a hand-written recursive-descent parser, when the example could have been solved with a regex.
Re: What follows from empirical software research?
#25One of the most important lessons I've learned in my career is that if a common problem has existed for a long time, a simple solution has probably been hiding in plain sight for a long time somewhere you haven't thought to look. I don't know anything about productivity research, but the fundamentals of defect density were figured out decades ago [1]. > (a) there’s no difference between test-first and test-after but…
Replying to my own comment to explain how it directly relates to the asked question - what to do with empirical research? The key takeaway is to evaluate everything in terms of sensitivity because it gives you design insight. The above quoted study identifies one insensitive factor (order) and one sensitive factor (iteration length) of interleaving test and code. Now you can choose, modify, or design any test method…
See, for me this is an example of how the hyper-empirical, sciencify-everything mindset is great for getting at truth but less great at giving advice.
The default, intuitive approach is to write a whole bunch of code and then test it. When people start doing TDD, they struggle with only making enough changes to the code to pass the test. But it helps a lot that the test suite tells them when to switch from writing code (which they prefer) back to writing tests (which they don't prefer) by passing.
Then we discover that the main reason TDD works is that it gets you to interleave coding and testing in small batches, and it works just as well to reverse the order of the batches but keep them small.
And then, somehow, knowing why TDD works (better than what people do by default) gets translated into "TDD doesn't work" (better than something that's carefully controlled to be exactly the same, except for the part that gets people to do the rest of it). And most of the people who hear that go back to writing a whole bunch of code and then testing it.
Because the real world doesn't control all the variables, so we often have to think about factors that the rigorous research is silent on because they can help or hinder us in achieving what the research says matters.
Re: What follows from empirical software research?
#26Earlier quoted context omitted.
Replying to my own comment to explain how it directly relates to the asked question - what to do with empirical research? The key takeaway is to evaluate everything in terms of sensitivity because it gives you design insight. The above quoted study identifies one insensitive factor (order) and one sensitive factor (iteration length) of interleaving test and code. Now you can choose, modify, or design any test method…
See, for me this is an example of how the hyper-empirical, sciencify-everything mindset is great for getting at truth but less great at giving advice. The default, intuitive approach is to write a whole bunch of code and then test it. When people start doing TDD, they struggle with only making enough changes to the code to pass the test. But it helps a lot that the test suite tells them when to switch from writing co…
As with any research, there are many possible audiences and it's not an article targeted at you. It's your job to already be educated in a way that's relevant to yourself.
The link I posted shows that the research has pretty comprehensively determined the relative impact of various factors, and I gave an example of how to interpret new research providing single data points within that framework. So now you should have the background that you need to read the literature with an eye for your application.
Re: What follows from empirical software research?
#27Earlier quoted context omitted.
The TDD example was just an example. I wasn’t taking the actual finding of the TDD research I was talking about a hypothetical finding. You can replace the examples with AIs like copilot and stipulate a large effect size. Nothing I said would really change. You still have to look at your desires to figure out what you ought to do given that research.
We're talking past each other, here. Your article presents a sufficient argument for dismissing the TDD example. Tossing effect size in, your argument still applies for dismissing an ideal study. My point is not that what you said didn't suffice, it's that the philosophically heavyweight arguments weren't necessary. They were a hand-written recursive-descent parser, when the example could have been solved with a rege…
Yeah, you could very well be right this is all overkill. But what I wanted to do was handle the general case rather than deal with things on a case by case basis. So to extend your analogy, I consider this blog post a parser generator. It isn't suppose to parse any particular text, but to allow you to create a parser for anything you want.
I've found that my short replies to empirical advocates seem to not connect. There seemed to be an assumption that I just wanted to hold onto my opinions and not think deeply about anything. So I wanted to do the exact opposite. Be as pedantic as I could (in the space of a shortish blogpost).