Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software engineering research is a train wreck

buttondown.email

Post reply on HN
Print view View original

41–50 of 174 posts

Re: Software engineering research is a train wreck

#41

Post by nonameiguess »

moksly wrote: post #29
SebastianFish wrote: post #6

Testing different methods of development in terms of speed, cost and quality is really hard. The most convincing approach to me would be a single blind experiment to hire two software development teams and have them build to the same set of requirements in two different ways. But then it is hard to know whether you are really comparing the method of software development or the quality of the software teams. So two so…

It’s not that hard is it? There recently was a study on the value of testing and best practices linked here on HN, that I of course can’t find now, where the researchers looked a thousands of projects. Over all there was no scientific proof that testing and best practices lead to better results than just making spaghetti without a recipe. Having worked in an public enterprise organisation that buys a lot of different…

A study design like that is called an epidemiological study and is far behind the gold standard of a random controlled trial, reason being the teams that choose to do testing or not are not randomly assigned to experimental and control groups. There are ways outside of study design of controlling for confounders when you can't randomly assign experimental and control groups, such as in this instance only looking at teams that directly tried similar projects with tests and without, but it is rare to see anyone do that.

Otherwise, you hit a rather obvious issue. Testing and following best practices are not the only policies impacting project quality, and in particular they exist in large part to help less experienced or hastily assembled teams. If you're comparing their output to the output of several core maintainers who have been working on the same project for 20 years, in the absence of other information, you expect the latter group to produce better quality work, and the fact that they actually do even if they aren't following industry best practices doesn't tell you those practices aren't useful to the former group or even that the latter group couldn't have produced an even better product if they'd followed them.

Be aware I'm not at all trying to advocate for either approach, just the issues with various flavors of scientific evidence that vary tremendously in how valuable they are depending on study design. I'm just saying we can't know with any level of scientific validity because the studies themselves are near worthless. Software management is in the state today that major league baseball was in 30 years ago, no statistically valid evidence and a whole lot of gut eye test from grizzled veterans. But unlike with baseball, nobody is keeping rich troves of every imaginable counting stat that can be counted going back a century on all of the developers, so a pure data science approach to making management more scientific like the moneyball guys accomplished in pro sports is not likely to work, since it would necessarily be data science without the data.

Re: Software engineering research is a train wreck

#42

Post by shusson »

> ...“bugs found in requirements are 100x cheaper than bugs found in implementations.” ... There’s one tiny problem with the IBM Systems Sciences Institute study: it doesn’t exist

It's funny to think how much of an effect this chart had on software engineering as a whole. I remember learning it at university and until today I thought it had some basis in science.

Re: Software engineering research is a train wreck

#43

Post by nonameiguess »

NohatCoder wrote: post #17
SebastianFish wrote: post #6

Testing different methods of development in terms of speed, cost and quality is really hard. The most convincing approach to me would be a single blind experiment to hire two software development teams and have them build to the same set of requirements in two different ways. But then it is hard to know whether you are really comparing the method of software development or the quality of the software teams. So two so…

"A specification that can be implemented using formal methods". That is just source code. If the specification can completely define arbitrary programs it is necessarily Turing complete on its own, and as such prone to the same type of bugs as any other program.

As soon as you need to interface with arbitrary external components, you see the value in good specifications. If JPEG-2000 was just a reference implementation and not a spec, that would work fine if only one team ever had to develop a JPEG library, and every application that read or wrote JPEG files used exactly that one library.

Since that isn't the case, having something sit at a higher level of abstraction than the actual source code is quite valuable. Additionally, it allows domain experts like image scientists and physicists, who are experts in how to compress and decompress data with minimal quality loss but may not be experts in any particular programming language, to still contribute to the spec.

Re: Software engineering research is a train wreck

#44

Post by MattGaiser »

star-trek-fleet wrote: post #26
jeffreyrogers wrote: post #12

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough,…

I happened to also delved into machining tools and work as SWE full time. This comparison to me, is not workable between machine building and software. So the difference is that machines do simple things and not changing. But software do complicated things and always changing. You can easily understand which parts and/or ways of operating that cause machine misbehaving. Because you have full grasp of the parts and ho…

> as long as they are machined according to spec, and torqued correctly during installation.

Isn't it more correct to say that nobody cares about screws because we have worked out the problems with screws to the point that we can trust they are machined to spec and torqued correctly?

Re: Software engineering research is a train wreck

#45

Post by ftio »

shusson wrote: post #37
ftio wrote: post #32

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about ar…

> Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. Do you think software engineering will remain a creative endeavour in the future? We've only been programming in high level languages for 60-70 years. All the while Moore's law has been in effect.

Oof. I have no idea. But since it'll be fun to guess:

Absent some kind of magical (from today's POV) AGI, I think I'd probably say yes, but I'm not sure that means software engineering will continue to look the way it does today, at least not universally.

My expectation is a pretty linear extrapolation of history. I think new tools and higher levels of abstraction will emerge that will make certain types of tasks unnecessary or a lot faster. We'll still need people working on compilers and embedded software, but a lot of really basic forms of software development (think Excel replacements) will be A LOT faster and easier to do.

Fundamentally, though, business problems are really fucking specific, and you need a really fucking specific language to express and solve those problems. The process of applying those languages to problems is a creative one. I don't think this dynamic will change for a really long time, if ever.

Re: Software engineering research is a train wreck

#46

Post by shusson »

jeffreyrogers wrote: post #12

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough,…

> It got me comparing precision machine design to software engineering.

I think the problem is that the failure of physical systems are often caused by random failures, which can be tested for and probabilities determined e.g how long before a bolt breaks. This as opposed to software, which most often fail due to systematic errors and cannot be tested completely.

Re: Software engineering research is a train wreck

#47

Post by indigochill »

jeffreyrogers wrote: post #12

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough,…

It's also difficult because software despite being a realm of abstract constructs still exists in the physical world. Hardware corruption can introduce bugs despite the software being formally correct. It's fascinating learning about hardware manufacturing for satellites and needing to build in hardware redundancy and error checking for bits getting flipped by solar radiation. And of course there was the famous rowhammer attack.

So you need not just software expertise but also hardware expertise if you're going to get serious about writing error-proof code, which Dan Bernstein happens to have.

Re: Software engineering research is a train wreck

#48

Post by pvarangot »

shusson wrote: post #37
ftio wrote: post #32

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about ar…

> Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. Do you think software engineering will remain a creative endeavour in the future? We've only been programming in high level languages for 60-70 years. All the while Moore's law has been in effect.

Music has remained pretty much a creative thing through most exponential explosions of technology that affected it profoundly. What's the most crazy futuristic scenario where software engineering stops being a creative field?

We have programming languages for creating and evolving organic lifeforms? You are basically creating plants, pets, "superworkers" or "super soldiers", or artists? That's going to still be creative. Like raising a child is creative. Same thing for stuff that can have an ego driven consciousness like most forms of AGI.

Re: Software engineering research is a train wreck

#49

Post by klysm »

IshKebab wrote: post #20

This tallies with my experience. It isn't early bugs that are expensive to fix. It's early design mistakes. For example choosing the wrong language, framework or architecture. Dropbox using Python is a good example. Or Python's GIL.

They aren’t really exclusive at all. Early design mistakes can also be really expensive, but a really scary bug that nukes all your data could also suck.

Re: Software engineering research is a train wreck

#50

Post by sanderjd »

shusson wrote: post #37
ftio wrote: post #32

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about ar…

> Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. Do you think software engineering will remain a creative endeavour in the future? We've only been programming in high level languages for 60-70 years. All the while Moore's law has been in effect.

I think doing the engineering design of structures and machines (buildings, bridges, engines, etc.) is still a creative endeavor after many hundreds of years, and I think this is the closest analogue to what software engineers do. The difference is that once we have the blueprints drawn (ie. the code written), a computer can just execute them rather than requiring a lot more material and labor inputs. Program execution is the right analogue to manufacturing, not program creation.
Post reply on HN