Software engineering research is a train wreck

In my field (mechanical), there are engineers, technicians, and drafters. Each of the above require more education than the next, and is paid better than the next. Someone who only does drafting work will not describe their job as engineering, nor are companies likely to ask engineers to do drafting because it doesn't make sense to pay someone an engineer's salary for that kind of work.

Basic static site seems like drafting. A CRUD app or WordPress site or similar between drafting and technician.

> But too often people forget [...] the value of qualitative metrics which can only be judged subjectively.

When I read the rant, I remembered a story about my university days and your sentence prompted my to write it down.

10 years ago I had to write a meta paper about Test Driven Development (TDD). I was researching this topic and found studies and other meta papers.

Some of the studies were from the likes of Microsoft where they explained they wrote two different drivers with comparable lines of code - one with TDD and one without - and tracked how many bugs they found after version 1.0 and how fast the driver projects were delivered. So in these non-trivial multi-month multi-person projects they claimed TDD was extremely useful to reduce bugs.

Other papers had 20 students learn how to program with TDD and another 20 students without and tried to find differences there. They couldn't find too many differences in the groups doing something like a 2 hour project. Since the group who just learned TDD was a little bit slower, they concluded neutral or negative value of TDD.

When I looked into the meta papers comparing these, I found some that were judging both of these papers as being of equivalent value, which to this day I'm really wondering about.

It is usually possible to do this (a digital journal can include links to cited/citing papers), but citations happen for a variety of reasons (background information, a single statistic, boosting a colleague's work...). Every citation applies to a different sentence or paragraph in an article. To understand the content and whether it is relevant, we still need to digest the text.

There are also less-cited papers and journals that can be just as relevant--every article starts with zero citations, and the vast majority of good work out there isn't exciting enough to be accepted for Nature.

Further, the process is rather incestuous: Previous person solved X problem with Y approach because that's the first thing someone came up with and it's just how it's done in our field, so let's continue doing this inefficient thing and citing that paper, maybe working on a better way to do things. Meanwhile in some other field, a mathematician or whatever came up with a far better solution to a similar problem long ago, but nobody in this field ever knew that, so it goes unnoticed.

I do want to see what is highly cited because it's probably interesting, but I also want to see things that don't get that kind of attention but are applicable to my work, and things that I wouldn't know are applicable to my work.

Another example: Currently, authors enter relevant keywords when they submit a paper. Maybe that works when someone searches the right combination of words, or maybe it doesn't because the search engines suck. Or because in one part of the world the topic has a completely different vocabulary, and we miss a whole library's worth of useful papers.

I speak from a background in STEM. I can't vouch for other disciplines, but I imagine they have similar pain points. Heck, I don't even know what others in STEM think, other than "that's just how research works."

I don't think it's an easy problem, or we wouldn't still be sifting through mounds of crap to find a few relevant, reproducible works worth reading and citing. I think it would involve figuring out the overarching themes and important methods in a paper and sorting them by their importance, and a little bit of fuzziness to say "hey, this isn't exactly what you're looking for, but it sure seems useful." This could even allow un-cited works a second chance.

I think I'm describing two separate goals, and the fuzzy part could wait until the relevance problem is addressed. I don't think the problems are particularly easy, or Google Scholar would have solved the problem and monetized the solution already. But it seems like they're solvable problems.

> I would say universally that bugs found prior to shipping are lower cost (not just cost to fix) than those found after.

There's a survivorship bias in this comparison, though. All software contains bugs - some are found in the architectural, coding, testing, review, and end-user stages. Bugs found after shipping are more likely to be significant to an actual user, because it's likely they were noticed in the course of someone using the software a particular way.

However, it's entirely possible that there are bugs that never need to be fixed. Perhaps these bugs involve scenarios that never crop up, or are part of a product direction that gets abandoned completely. (Example: Quibi probably didn't need to handle credit card expiry gracefully, that would have been wasted effort).

Those bugs that made it into prod and got prioritized for a fix, and were memorable are likely to be higher effort just by the nature of those filters.

It seems to go something like this every time:

* We create new technology.

* There's an explosion of creativity and applications of the technology.

* This leads to an explosion in the personalities involved.

* Some of these gravitate towards the tedium and expound on it constantly.

* Some person or group automates that away and increases the ability to be creative.

* Repeat from the top.

It's always fun to watch the wheel go around though. Never any shortage of commentary on HN about it!