Software engineering research is a train wreck • Hacker News Board

Re: Software engineering research is a train wreck

#111

Post by marcosdumay » Tue, Jul 20, 2021, 9:36 PM UTC

krisoft wrote: post #83

eschneider wrote: post #55

Earlier quoted context omitted.

It "feels right" and is therefore "truthy."

Do you guys doubt that it is right? What kind of information would you require to show you that it is true? I'm one of the lucky ones who tends to work on safety critical "complex cyber-physical systems". Is that maybe the difference? There is the case where a developer runs unit tests before a commit. Some of the tests are unhappy. Developer fixes it and commits it up. Maybe hours spent? Then there is the case where…

The chart very likely has the correct direction, and the numbers are probably completely wrong. From another point of view, the literal phase division there is nonsensical, but the idea it communicates isn't.

But that's not relevant. What's relevant is that this is not how you do research. You can't just assume that. Yeah, there are plenty of examples of high-cost production failures. There are plenty of examples of low-cost production failures too, and plenty of examples of projects that failed because people spent all their time fixing the same few issues. If you want to state it as a fact, you have to count all of those and see how they compare.

Re: Software engineering research is a train wreck

#112

Post by hyperhopper » Tue, Jul 20, 2021, 9:38 PM UTC

mcguire wrote: post #95

" Here’s the only technique I’ve found that works, which I call scrobbling even though that means something totally different: " 1. Search seed terms you know, like “cost of bugs”, in an appropriate journal (here’s one). " 2. Find papers that look kinda relevant, skim their abstracts and conclusions. " 3. Make a list of all the papers that either cite or are cited by these papers and repeat. " 4. Find more useful ter…

The problem stated is that instead of being able to find widely cited or good papers, any amount of research involves trudging through hundreds of bad, off topic, outdated, unrelated, or otherwise useless papers, just to get an idea of what is being talked about in the field. And even when getting there, what you find may be completely non-practically relevant in the end.

Just saying "yeah thats how things work in academia" Doesn't help anything; this article is pointing out a very real problem: the state of things now is a mess.

Re: Software engineering research is a train wreck

#113

Post by likeabbas » Tue, Jul 20, 2021, 9:46 PM UTC

stonemetal12 wrote: post #106

jeffreyrogers wrote: post #12

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough,…

> Rather than an error budget, you'd have some sort of bug budget, where you tracked what sort of bugs were caused by what sort of constructs, and design your program in such a way to minimize their chance of being introduced. Aren't we more or less doing that at the language level? C had lots of memory bugs, so C++ tried to fix it with collections, iterators, and smart pointers. It was still hard, so Java came along…

This would make sense if we migrated existing software projects to different languages.

Re: Software engineering research is a train wreck

#114

Post by osigurdson » Tue, Jul 20, 2021, 9:57 PM UTC

musicale wrote: post #70

ftio wrote: post #32

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about ar…

> Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process Exactly. And it's less like movie production and more like 4,000 people trying to collaborate to produce a million-page novel. It does bear some resemblance to design and engineering, but with custom materials and components that have never been used before and need to be created specifically for the proje…

Generally, analogies to software engineering are not helpful.

Certainly, software engineering may be a little like problem domain A, B and C but probably less so than we think. After all, how acute is our knowledge of these problem domains to begin with?

Re: Software engineering research is a train wreck

#115

Post by lifeisstillgood » Tue, Jul 20, 2021, 10:00 PM UTC

hwayne wrote: post #87

lifeisstillgood wrote: post #79

My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. Software is more like a novel than a bridge. Maybe, maybe we can apply engineering terms to machine code or assembly. But the abstract levels we all wor…

> My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. 1. This is false. One of the reasons making bridges is so expensive is because each bridge is a precious special snowflake with lots of challenges spe…

1. I agree bridges have special challenges, but ... well for 2,000 years there was a keystone, and an arch. Yes the soil and the land were challenging, but I find it hard to argue that every stone bridge was a special snowflake. Snap a supporting cable on any suspension bridge and they all have the same failure mode. I think we are stretching analogies - is "build a CRM for the sales department" simple as in "there is a keystone data design common across all CRMS - and all the rest is "shifting sands of the river are same as shifting alliances in the boardroom"

I just try to keep this simple. Coding uses symbolic representation to describe a model - this is what language is. Compilers do / can do things that simply don't exist conceptually in Engineering.

Coder manipulate symbols. It really is a language thing.

2. yes. I was just using bridges as an example of engineering. As CRUD websites are an example of software engineering.

Re: Software engineering research is a train wreck

#116

Post by [deleted] » Tue, Jul 20, 2021, 10:11 PM UTC

jeddy3 wrote: post #75

ftio wrote: post #32

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about ar…

I seems like many posters in this thread try to classify software enineering as either creative or "mindless factory-work". Where actual enineering disciplines has the risk of removing the creative part. I think this classification is wrong. There IS NO mindless factory-work. Just as in other enineering disciplines, our work is not manufacturing. It's just that the actual manufacturing does not exist (or rather is do…

[deleted]

Re: Software engineering research is a train wreck

#117

Post by aeturnum » Tue, Jul 20, 2021, 10:12 PM UTC

Software engineering research is immature and, as such, has few (no?) strongly-supported claims that the entire field relies on. I agree this is a frustrating state of the field, but the way this is written makes the author appear to be unfamiliar with the state of things.

For example:

>Did I mention that all three of those papers use different definitions of “defect”?

This kind of thing is really common in academia. I do not know software engineering research in detail, but there are older and more basic definitional fights in the social sciences. Defining a defect, as I'm sure the author of the blog would agree, is not a trivial decision.

In general, this piece contains (perhaps hyperbolic?) dismissals of people trying to do work. It doesn't seem to respect what I imagine is earnest effort. Taken all together, it seems like the author either thinks the entire field is charlatans (which I doubt as they use the fields' conclusions at the end) or is making light of people working on problems they themselves do not have answers to in order to have a more lively writing style. It was fun to read and left a bad taste in my mouth.

Re: Software engineering research is a train wreck

#118

Post by Joeri » Tue, Jul 20, 2021, 10:18 PM UTC

I went into this software engineering productivity research rabbit hole a decade ago and came away similarly depressed with the lack of rigor in our field.

What I’ve come to believe:

- Software engineering is not an engineering discipline, it is a craft. It may or may not one day become an engineering discipline, I find this impossible to tell.

- The difference between a feature and a bug is largely one of semantics, not of substance. A defect is a mismatch between expected behavior and actual behavior. A bug is when the expectation was that of the programmer that wrote the code. A feature request is when the expectation was that of a user. There is no real difference for the user between a missing feature and a broken feature. Hence why we so often get side-tracked into fruitless “bug or feature” discussions and why users don’t care when we say it is by design.

- Errors will happen. People are flawed and produce errors at a more or less set average rate for each person (unless they’re tired or enjoying a ballmer peak). Errors in every step of the process, from design to deployment, are inevitable. We can however catch these inevitable errors, with automated error detection tools, or by having a fresh set of eyes look at the work. The consequence is that making high quality software necessitates taking a look at the whole process and introducing error detection steps in every part of it, whether they be automated like linters and unit tests, or human like design and code review or mentoring and pair programming.

- Errors in design are more costly because they cause entire features to need to be written or rewritten, which is the most costly kind of defect to resolve. We do not need research to know this to be true. We solve this by having short feedback loops with real users, which is the only real form of agile development.

Re: Software engineering research is a train wreck

#119

Post by JimTheMan » Tue, Jul 20, 2021, 10:18 PM UTC

ska wrote: post #23

jeffreyrogers wrote: post #12

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough,…

> while no large team (that I'm aware of) has done anything similar. In areas like avionics, medical devices etc., there are formal production engineering methods applied to software as a matter of course. It can very work well, but it is definitely expensive compared to industry average.

I have my suspicion that medical devices aren't as rigidly programmed as the general public would like. Ie programmed by a biomedical/electrical engineer who isn't really a software specialist. But I'd be keen to hear from someone in the trenches.

Re: Software engineering research is a train wreck

#120

Post by lolinder » Tue, Jul 20, 2021, 10:26 PM UTC

lifeisstillgood wrote: post #115

hwayne wrote: post #87

Earlier quoted context omitted.

> My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. 1. This is false. One of the reasons making bridges is so expensive is because each bridge is a precious special snowflake with lots of challenges spe…

1. I agree bridges have special challenges, but ... well for 2,000 years there was a keystone, and an arch. Yes the soil and the land were challenging, but I find it hard to argue that every stone bridge was a special snowflake. Snap a supporting cable on any suspension bridge and they all have the same failure mode. I think we are stretching analogies - is "build a CRM for the sales department" simple as in "there i…

> I just try to keep this simple. Coding uses symbolic representation to describe a model - this is what language is. Compilers do / can do things that simply don't exist conceptually in Engineering.

> Coder manipulate symbols. It really is a language thing.

What is conceptually different here between software engineering and civil or mechanical engineering?

Engineers use symbolic representations (math, diagrams, computer models, etc.) to describe a model of what will eventually be manufactured or built in the real world. They're typically not the people who actually build the thing, they're the ones who describe it in sufficient detail that someone else can build it. This is as much a language problem as it is in software. The designs need to be readable by the entire team of engineers as well as the people who will make the design real.

The primary difference I see is that in software engineering, the "someone else" who will build the thing is a computer program called a compiler rather than a factory worker or a construction crew.