Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What follows from empirical software research?

jimmyhmiller.github.io

Post reply on HN
Print view View original

11–20 of 27 posts

Re: What follows from empirical software research?

#11

Post by dasil003 »

kqr wrote: post #10
dasil003 wrote: post #3

> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical re…

> That said, I don't believe that statistically significant results can be found that will overturn my own instincts and judgement on any specific project to which I am dedicated. The reason for this is threefold: 1) the universe of software and goals we pursue with is astronomically large 2) competence in software engineering depends on the combination of personal aptitudes and mindsets combined with years of practi…

Word-for-word? Really? Sorry, I just don't understand your point.

Are you saying merchants in the 1700s didn't believe insurance outcomes were quantifiable? Or are you saying that software engineering output is quantifiable? If the latter, maybe you might shed some light on how you think that would work, I'm happy to be proven wrong.

Re: What follows from empirical software research?

#12

Post by nunuvit »

One of the most important lessons I've learned in my career is that if a common problem has existed for a long time, a simple solution has probably been hiding in plain sight for a long time somewhere you haven't thought to look. I don't know anything about productivity research, but the fundamentals of defect density were figured out decades ago [1].

> (a) there’s no difference between test-first and test-after but (b) interleaving short bursts of coding and testing is more effective than working in longer iterations

I'm glad you quoted this study because it's a perfect example of a conclusion that should have been the starting point of a more interesting experiment. Interleaving code and test is known to be the most impactful factor. TDD and TRL don't differ in any of the ways that account for the vast majority of defects. Therefore of course the difference should be small, and of course shorter iterations should be better.

[1] https://www.slideshare.net/AnnMarieNeufelder/the-top-ten-thi...

Re: What follows from empirical software research?

#13

Post by riwsky »

jimmyhmiller wrote: post #6
riwsky wrote: post #5

The author's overthinking it. He cares about productivity—it's just that the effect sizes are too small in these studies to overcome his prior beliefs.

In the article I’m assuming ideal conditions. So if you think a large effect size is important, throw that into the ideal conditions. I don’t think that changes anything I wrote. Maybe I’m missing something?

If you throw that in, you’d need to throw the TDD example out. A stronger test of your arguments would be, eg, research on productivity impact of generative AIs like copilot.

(This would grow your example to match your argument, whereas my initial post would shrink your argument to match your example.)

Re: What follows from empirical software research?

#14

Post by zmmmmm »

I obviously have not looked at all (or really much) of this research, but I have always felt that software development is so context dependent that drawing generalised conclusions is just an impossible ask in the first place. Even if you did, there would be so many exceptions that in practice it will come down to "use your experience to assess the context and then decide".

The example of TDD : I've done significant pieces of work both with and without TDD. In some scenarios its a huge impediment; the actual complexity of the internal software is fairly low and the but the testing complexity is high (many complex stateful dependencies that are hard to control). In that case, I spent 80% of my time writing the tests and far more bugs surfaced in the tests than the code itself.

Then in other scenarios there's high internal complexity, low external dependencies / complexity and it's pretty much a no-brainer, TDD is almost the only tractable way to write the code let alone an improvement.

Then it's very personal as well. One person will work well with TDD and another will struggle. Dumb things like, is your personal preference in development environment conducive to rapidly running and iterating on tests are probably going to dominate.

End result is, I think these studies just can't possibly control all the variables and this is why they either end up in invalid conclusions, too specialised conclusions, or, as Jimmy says, the more rigorous the study the less significant the result are.

Re: What follows from empirical software research?

#15

Post by felipefar »

dasil003 wrote: post #3

> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical re…

Do you think methodology in general isn't useful or just empirical software research?

Methodology learns from the experience of the experts, and tries to teach the techniques they know to beginners. It's very different from statistics.

Re: What follows from empirical software research?

#16

Post by jimmyhmiller »

riwsky wrote: post #13
jimmyhmiller wrote: post #6

Earlier quoted context omitted.

In the article I’m assuming ideal conditions. So if you think a large effect size is important, throw that into the ideal conditions. I don’t think that changes anything I wrote. Maybe I’m missing something?

If you throw that in, you’d need to throw the TDD example out. A stronger test of your arguments would be, eg, research on productivity impact of generative AIs like copilot. (This would grow your example to match your argument, whereas my initial post would shrink your argument to match your example.)

The TDD example was just an example. I wasn’t taking the actual finding of the TDD research I was talking about a hypothetical finding. You can replace the examples with AIs like copilot and stipulate a large effect size. Nothing I said would really change. You still have to look at your desires to figure out what you ought to do given that research.

Re: What follows from empirical software research?

#17

Post by sdwr »

kqr wrote: post #10
dasil003 wrote: post #3

> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical re…

> That said, I don't believe that statistically significant results can be found that will overturn my own instincts and judgement on any specific project to which I am dedicated. The reason for this is threefold: 1) the universe of software and goals we pursue with is astronomically large 2) competence in software engineering depends on the combination of personal aptitudes and mindsets combined with years of practi…

"Is success due to simple rules applied rigorously, or individual heroics?"

Yes!

Re: What follows from empirical software research?

#18

Post by marcosdumay »

What a lot of words to justify continuing preaching TDD despite no evidence that it's better. (Guess what, it's not worse either, so if you want to personally use it, go for it. Just stop insisting other people should "convert".)

Of course if something has a huge impact on my productivity, I want to practice it. Even if it's not fun. There's a lot of denial embedded on this article.

Re: What follows from empirical software research?

#19

Post by jimmyhmiller »

marcosdumay wrote: post #18

What a lot of words to justify continuing preaching TDD despite no evidence that it's better. (Guess what, it's not worse either, so if you want to personally use it, go for it. Just stop insisting other people should "convert".) Of course if something has a huge impact on my productivity, I want to practice it. Even if it's not fun. There's a lot of denial embedded on this article.

I think you missed the point of the article. I actually don’t do TDD at all. I’m actually personally not a fan. So feels a little weird that you thought this was an article trying to justify TDD use.

Re: What follows from empirical software research?

#20

Post by rTX5CMRXIfFG »

zmmmmm wrote: post #14

I obviously have not looked at all (or really much) of this research, but I have always felt that software development is so context dependent that drawing generalised conclusions is just an impossible ask in the first place. Even if you did, there would be so many exceptions that in practice it will come down to "use your experience to assess the context and then decide". The example of TDD : I've done significant p…

It seems to me that you misunderstand the purpose of statistical generalizations in the first place? The purpose of looking at multiple cases is to derive common patterns regardless of the differences in detail among individual cases. Which is why, yes, you still need to apply the inferences into a specific context if you need to make a decision, but to say that nothing can be gained from a comparative big-picture approach is a malicious withdrawal from critical thinking.
Post reply on HN