Ask HN: How to transition from academic programming to software engineering?

I tend to come down on your side of the craft vs. engineering debate, but then I disagree with basically your entire argument for it :)

You list three distinguishing features of an engineering discipline. The first and third swap cause and effect; a field doesn't become engineering because once it requires licensure and years of study. Surely you wouldn't agree that a successful campaign to require those things for software developers would bestow the status of engineering upon the work.

Your second point seems closer to the truth, but I'm not so sure it's true. If someone comes to a civil engineering firm and says, "design us a road between this city and that city", that is often a unique challenge because the terrain between those two cities is likely unique, maybe it requires a big bridge or a tunnel, which will be akin to but not identical to other bridges and tunnels you've built before.

Recently I've been involved in transitioning an academic software piece to an open source library. One of the most noticeable things is the different priorities and emphasis on what is driving value in these different environments. The people who were making the code before had priorities mostly to do with research, the main artifacts were papers and research, the software itself was not the main artifact. The interesting thing is that they had good software and research skills so it wasn't a matter of bad skills muddying the waters and hence gave a great spotlight into how different people can have different priorities with code. So when we were making it into a library which others could base their work off there was a big shift in priorities because the code became an artifact worthy of directly spending more time/money on. You may find what we wrote about this process interesting as it highlights the things from a software engineering/open source perspective that were now important and had to be done to make the project a standalone library useful for consumption by other developers: https://www.customprogrammingsolutions.com/blog/2018-02-25/P...

Software development as practiced now by a huge number of individuals and companies is closer to a craft, but it can be and must be more than that if we want to be able to tackle the growing complexity of software and improve its overall barely adequate quality.

Crafts don't scale and are a poor fit for highly complex domains.

The curse of software development is its huge financial success, anemic legislative specification and the observed reality that customers will still buy poor quality software.

These are preventing the craft-like programming from turning into software engineering, but the craft is already failing to reach expectations: countless security disasters, unethical programmers enabling spying on millions, software literally killing users. This stuff will only get worse.

And finally, we do understand software engineering well enough to teach it properly. It's just not done, because it's not considered necessary when one can get by with a computer science degree, no degree or a bootcamp certificate.

"The first and third swap cause and effect;"

Sorry, I wrote that in a hurry. I wasn't claiming either was a cause or effect. It was more of finding characteristics that we can use to identify one from another. I.e. following the argumentation "If it quacks like a duck and walks likes a duck it's likely a duck, and if it doesn't, we don't really have much evidence of the duckish quality of the observed thing".

So, I was not aiming to claim that licensure would turn software engineering into actual engineering. Rather, that the requirements of the field are so poorly understood in the general context that there would be very little to agree on the specific requirements. Poorly understood -> not engineering, really.

I totally agree with what you wrote above.

On the third point: I'm not claiming 100% truthiness to my argument, but it's pretty close. Software engineering projects are still among the riskiest ventures where you can think of investing capital in. If you want to build a road:

1. The language of the requirements are pretty well understood, from point A to B, this many lanes 2. Unless some unforeseen calamity arises, and you have the capital to pump to the project, eventually you will get the road

I think we can agree that these define any engineering project. Of course, engineeering is not cut and dried either - that's why you need to have actually trained professionals who can react to the events that come along as the project progresses.

I don't think 1. or 2. can hold for a software project in the general sense. Furthermore, you can end up accidentally, without anyones fault, wasting an arbitrary amount of capital on a feature that could, in the worst case, be replaced by a few lines of Python.

This poor quality of our general understanding of software development and lack of common language to describe anything means that most of the time software develoment is closer to R&D than engineering.

Generally, you can get better estimates when you are implementing a similar project the nth time. Like some general website, or a server backend. And in these instances you have the language to describe features and requirements. But in the general sense, software development isn't anything like this.

"When we say software development is a craft, we're saying that it's like shoemaking, pottery or woodworking."

The point being, there are intricate details that are very hard to deliver in the traditional class-room oriented school environment with well defined requirements.

Those do not state anything about scalability. Crafts can scale - like for example how the old giant cathedrals and castles were built in middle-age europe.

They don't say anything about mindsets either. You need an engineers mindset to buid a cathedral or a castle.

The specific problem with crafts is that adapting to new requirements is a complete hit and miss process. The reason for this problem is the lack or proper theoretical framework in which to pose ones work and into which embed the requirements themselves.

The program verification people are working towards solving this problem, see for example Leslie Lamport's work in TLA+.

But until we have a general, mathematical proof backed compiler for requirements, as well as for the program implementation, we are pretty much stuck with craftsmen.

(Well, we have proof compilers but those are at the moment completely unusable for general programming since they are so complex to use.)