What follows from empirical software research?
jimmyhmiller.github.io
What follows from empirical software research?
1–10 of 27 posts
Re: What follows from empirical software research?
#2Does any software engineering research take into account the human factors of context, interest, exhaustion, and aptitude?
Re: What follows from empirical software research?
#3> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors
As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical research has to say. I think it's important to always remain open to good new ideas wherever they may come from—strong opinions, loosely held, as they say.
That said, I don't believe that statistically significant results can be found that will overturn my own instincts and judgement on any specific project to which I am dedicated. The reason for this is threefold: 1) the universe of software and goals we pursue with is astronomically large 2) competence in software engineering depends on the combination of personal aptitudes and mindsets combined with years of practice and 3) measuring outcomes in software engineering across diverse projects is all but impossible. In other words, you can't equate tools, you can't equate projects, and most of all you can't equate people.
At the end of the day, success in software engineering comes from relentless focus on the specific goals at hand. One must be inherently curious and have a craftsperson's mentality about acquiring technical skill, but never become religious about methodology. This requires continuous first-principles thinking targeted at specifics. At the end of the day, two expert practitioners could propose unorthodox and diametrically opposed approaches to the same problem, and they would still dramatically outperform a lesser skilled journeyman who attempted to follow every best practice.
Empirical studies and the scientific method in general work fantastically well for uncovering the rules and inner workings of the natural world, but software is the creation of logical systems purely by human minds which is an entirely different challenge—there's just not enough evidence to draw on. I suspect results will be at least a couple orders of magnitude softer than sociology, and that probably won't sit well with the type of personality attracted to software in the first place.
Re: What follows from empirical software research?
#4For personal software development projects of course there are other factors that matter beyond finding the theoretically optimal X and Y. From a management perspective in business those factors might also matter. You want to get the best performance out of your team but you're not going to do that if people keep quitting due to an unpleasant work environment.
As far as advocacy goes though - when someone is recommending what other people should do - I think it's very different if there is relevant evidence and it doesn't back up the advocated position. It's even more different if there is relevant evidence that positively undermines the advocated position. There are snake oil salesmen in this industry and some of them will call you names if you don't follow their pet process. But if what they're peddling isn't backed by the evidence or even contradicts the evidence then they should be called out and their audience should probably be sceptical about anything else those same salesmen are selling as well. The old joke about someone finding it hard to believe something when their continued employment depends on its falsehood is as relevant as ever.
Re: What follows from empirical software research?
#5The author's overthinking it. He cares about productivity—it's just that the effect sizes are too small in these studies to overcome his prior beliefs.
Re: What follows from empirical software research?
#6The author's overthinking it. He cares about productivity—it's just that the effect sizes are too small in these studies to overcome his prior beliefs.
In the article I’m assuming ideal conditions. So if you think a large effect size is important, throw that into the ideal conditions. I don’t think that changes anything I wrote. Maybe I’m missing something?
Re: What follows from empirical software research?
#7Does any software engineering research take into account the human factors of context, interest, exhaustion, and aptitude?
i doubt it. Similarly to the project trinity - functionality, price, time - that never included Fun ..
Re: What follows from empirical software research?
#8> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors; whatever it is that we need to proclaim it to be a good study.
Has there ever been an empirical software study, that would have a beautiful research plan, sound statistical analysis, large sample size, and that would also have been replicated? Even one?
Re: What follows from empirical software research?
#9"Empirical software research" could mean a bunch of different things. This article is about studying people writing software, not about software used for research in empirical sciences, and not about research into computer science.
I'm confident the answer to what follows from that is "nothing yet" based on various conference talks. Studying developers (or in a worse case students) writing software doesn't seem to be an effective way of working out how to write software better/faster/whatever.
Re: What follows from empirical software research?
#10> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical re…
> That said, I don't believe that statistically significant results can be found that will overturn my own instincts and judgement on any specific project to which I am dedicated. The reason for this is threefold: 1) the universe of software and goals we pursue with is astronomically large 2) competence in software engineering depends on the combination of personal aptitudes and mindsets combined with years of practice and 3) measuring outcomes in software engineering across diverse projects is all but impossible. In other words, you can't equate tools, you can't equate projects, and most of all you can't equate people.
This was pretty much the word-by-word argument against using statistical approaches to price insurance of shipments over the sea back in the 1700s. Yet we all know how insurance premiums are calculates today, and there's a reason for it.