Software engineering research is a train wreck
buttondown.email
Software engineering research is a train wreck
1–10 of 174 posts
Re: Software engineering research is a train wreck
#2If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
Re: Software engineering research is a train wreck
#3I generally agree with everything, but thought I would just use this as a springboard into a related topic:
> What’s the difference between a “bug” and “defect”? Well, one’s a bug and the other’s a defect, duh!
This kind of issue is common but I'm not sure how to avoid it. Any group of people > 1 will start to use their own lingo, which often is made up of similar words from "outside" but have different connotations. This is true in science, medicine, software engineering, law, everywhere.
(I mean, why would I search for "bug"? Like, I'm searching for problems with computer code, not insects!)
This phenomenon unfortunately leads to misunderstandings with the general public, which leads to mistrust. Part of it is on scientists (and lawyers...) to be clearer in their communication, but I think it is also on the public to recognize that when reading scientific literature they are not the intended audience and are therefore missing a ton of context that is not explicitly stated.
Also:
>I’m sure this is slightly easier if you’re deeply embedded in academia
Also depends on the field. Chemistry has SciFinder, which although very expensive for institutions, is very good. It is fairly specific to chemistry though (and some overlapping fields).
His process for finding node papers and grinding through citations is pretty much how most scientists do it, though. And conferences.
Re: Software engineering research is a train wreck
#4If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
There's another strategy that works better than searching for a keyword, which is to read the article.
It's about the difficulty/expense of systematically and scientifically studying the software development process (e.g. finding out what data actually exists to support the claim that "a bug is cheaper to squash when it is caught during the specification process").
The title expresses frustration not at science itself, but at how needlessly difficult this process is, since both the secondary and primary academic literature surrounding it are of extremely low quality.
Re: Software engineering research is a train wreck
#5> I’ve checked a few other papers and think I’m tentatively confident in this line of reasoning: certain bugs take more time to fix (and cause more damage) than others, and said bugs tend to be issues in the design. I haven’t vetted these papers to see if they don’t make major mistakes, though. It’s more time-consuming to do that when you’re trying to synthesize an stance based on lots of indirect pieces of evidence. You’re using a lot more papers but doing a lot less with each one, so the cost gets higher.
Empiricism and quantitative metrics have indispensable value, that much should be clear to everybody I hope. But too often people forget (or have active contempt for) the value of qualitative metrics which can only be judged subjectively. Such considerations are naturally harder to deal with than cold hard data, so it doesn't surprise me that people want to avoid it. But when you blind yourself to the qualitative and subjective, you do yourself a huge disservice. Just ask McNamara; he thought he could win the Vietnam War with quantitative metrics and utterly neglected difficult to quantify metrics like public sentiment, both in Vietnam and America. I see echos of this in our industry today; we love to talk about empirical measures like the number of bugs, but subjective metrics, like the severity of those bugs, receive less attention.
Many university programs are set up to address this, by making engineering students earn credits in the humanities as well. But I fear the value of this is often inadequately explained. Contempt for the humanities and scientism go hand-in-hand, and are a worrying trend particularly in the tech industry.
Re: Software engineering research is a train wreck
#6Testing different methods of development in terms of speed, cost and quality is really hard. The most convincing approach to me would be a single blind experiment to hire two software development teams and have them build to the same set of requirements in two different ways. But then it is hard to know whether you are really comparing the method of software development or the quality of the software teams. So two software teams isn't enough to get a statistically valid inference. You can see that, given software development rates, this could become a very expensive experiment.
Last point. I think that even writing a specification down to the level that it could be implemented using formal methods might be the biggest game changer. Agile stories rarely come even close to covering all of the potential edge cases. If we had a process that required product owners to literally think through all possible failure modes (what systems of formal methods do) and write out how to handle them then the cost of writing specifications would go way up. Per economics, I think we would end-up with simpler specifications which might be its own benefit.
Re: Software engineering research is a train wreck
#7If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
There's another strategy that works better than searching for a keyword, which is to read the article. It's about the difficulty/expense of systematically and scientifically studying the software development process (e.g. finding out what data actually exists to support the claim that "a bug is cheaper to squash when it is caught during the specification process"). The title expresses frustration not at science itsel…
But specifically with respect to software engineering, it's really hard to do solid, repeatable experiments. It's not like physics, it's like sociology or psychology - which also have trouble doing precise, repeatable science.
Why is it like sociology or psychology? Because it's about people, not about things. Software engineering is not just about languages and programming techniques; it's about how people interact with those languages and techniques. The people have far more variability than the languages and techniques do. Cutting through that to be able to accurately say something about any language or technique is... really hard.
Re: Software engineering research is a train wreck
#8If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
Try figuring it out through the context. There are paragraphs upon paragraphs of it in the article.
Re: Software engineering research is a train wreck
#9It's completely besides the point, but the capitalizing of the I of « **Ing » in the title makes me unreasonably grumpy.
Re: Software engineering research is a train wreck
#10If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
There's another strategy that works better than searching for a keyword, which is to read the article. It's about the difficulty/expense of systematically and scientifically studying the software development process (e.g. finding out what data actually exists to support the claim that "a bug is cheaper to squash when it is caught during the specification process"). The title expresses frustration not at science itsel…
Despite insistence of programmers wanting to be scientists and engineers, it's not true to the definition.
Maybe safety critical C or assembly is engineering.
Maybe testing code for speed can be considered science.
By the time any useful program is finished, there's significantly more tradition and art than Science and engineering.