A New Software Engineering

Back in the day, craftsmen were just people who were so specialized in a trade that they could build amazingly complex and difficult things through example and practice. But it took a long time to learn this craft from another craftsperson, and knowing this one trade so well left them at a loss for other aspects of the thing they built (resulting in things like building collapses).

Modern software developers are the same. Indeed, we're even going through the cultural shift that happens when different civilizations revisit the same things without really looking at how their forebears did it. We're still reinventing the wheel instead of creating a better one, and we're far from creating any new transportation mechanisms.

In order to achieve the kind of evolution from 'craftsman' to 'engineer' that existed for physical architecture, software developers need to learn about things other than software. It's not enough to simply learn how the kernel works, or the hardware works. You have to learn how all those pieces work with other pieces, and the resulting interactions between different natural and non-natural processes with computer systems.

The author gets to those points with his 'essence' of new software engineering. But it gets a bit bogged down by not being generic enough, and not being specific enough. It's simple to see the difference between a craftman's output and an engineer's: the application of scientific principles to achieve an output we can be much more confident in. And that's what 'new software engineering's goal should really be: producing a more reliable, reproducible, safer product.

Are the proposed methodologies going to get us there? I don't think so. I think we need less process, and more science, and to create software which has science as an inherent requirement of its design and implementation, and not merely an afterthought for performance reasons.

Finally, we'll be able to write distributed low-latency software that interacts with both legacy systems and browsers with complete end-to-end type safety, provable correctness and fault tolerance! In half the time, at half the cost!

With spider charts.

The first is the operating system kernel in the air link processor of mobile phones. In most current phones, that's an L4 kernel with a full proof of correctness. Since any mobile phone can potentially knock out all phones for some distance around if it doesn't follow the sharing rules for the air link, this is important to carriers. They got it right. Nobody talks about this much, but if that layer had problems, there would be regular cellular blackouts.

The second is the Windows Static Driver Verifier. This has been used since Windows 7. It verifies that kernel drivers don't crash, clobber memory or call the driver APIs incorrectly. Before the Static Driver Verifier, drivers accounted for more than half of Windows crashes. Now, crashes from signed drivers are very rare, and usually involve getting the driven device itself to do bad DMA operations. (IOMMUs are coming along to stop that.)

This shows the right direction for software engineering. Some software really matters, and has to be engineered properly. Most software doesn't matter all that much. Engineered systems should separate the two, develop them in different ways, assume the low-grade stuff will crash, and architect systems so the low-grade stuff can only do limited damage. We're seeing architectures like that in the mobile world and in server-side systems. In the mobile world, "apps" run in relatively contained environments. In the server world, things seem to be moving towards containerized "apps" in systems like Docker, running on some minimal glue layer inside a container running on a secure microkernel such as Xen and talking via message passing.

That's software engineering.