The only build system that might someday replace make (2010)

In the subsequent decade that this tool fell short of world conquest, containers and CI/CD pipelines have grabbed much mindshare.

The question is less: "How do we derive a simpler tool?" and more "How do we convert enough of the prior art to hit a tipping point?"

Bazel is about as far from redo as you can get: they aim to solve very different problems. If you have a huge monorepo, and thus need distributed caching and distributed builds, and cross-language dependency tracking and incremental rebuilds, Bazel is for you. It has the advantage of being built _after_ Google understood all the places where lack of rigor or accidentally-too-capable build language features cause trouble.

We’ve been using it to build and deploy a ~500k loc JavaScript monorepo and I have an unpopular opinion about it. While I see the benefits of it for google scale, for us it’s just been one headache after another.

Reimplementation of python for build file language? No thanks.

Reimplementation of package managers? No thanks.

Reimplementation of Docker? No thanks.

It’s just another thing to learn, worry about, and discover bugs in rather than something that just works. I already know how docker, yarn, and python work and that stuff works well, why are we boiling the oceans recreating and relearning all of this just to do it the “Bazel” way?

What a waste of time and energy!

Could you talk a bit about your experience? The two points you mentioned feel more like theoretical / principled opinions rather than on the ground experience / issues

Ohhh the tragedy of build systems... Maybe the best example of meta technical debt on the planet, for so many canonical reasons that technical debt emerges in general. Top reasons no software engineering team fixes this:

The build system will not directly improve the end product. So no company wants to pay for it / use the time.

Due to the previous reason, a build system is not a compelling value proposition as a product (except for speed, see next)

Hardware/cloud technology has progressed fast enough that raw horsepower can be thrown at a slow build system with reasonably good results.

The ubiquity of legacy build systems makes integrating anything new extremely painful. Almost no software project these days is free from non-trivial dependencies that will certainly be using a legacy build system. Again, for any project team, this pain and extra effort cannot be justified as it doesn't make a difference to the end customer.

Engineers hate arbitrary syntax changes with respect to some scripting language when there is equivalent common practice. This goes for new general purpose languages too (entirely separate rant, but what is the point of arbitrarily changing the most widely known, generic C-style syntax of a function call? What value does it provide not to just re-use the return_type name(arguments) pattern that 99%+ of all programmers must already know? What is the return on the cognitive load???)

Most engineers simply don't care. They aren't interested. They click a green arrow or type "make" as a ritual and then wait for what they care about, which is the end-result. The only time anyone cares is when it has to be setup for the first time, or when it breaks. Everyone at this time will moan and say "this could be better, this build system sucks, we should make a better one", but once it is setup/fixed, everyone rapidly forgets about it never to return.

It is not a sexy problem. Telling anyone outside of software engineers that you made this awesome build system is pointless - they won't even understand the premise of the problem. In fact, interviewing recent CS grads, even they don't really know what a build system is.

  - If you think about it for a moment, these sort of reasons are the harbingers of technical debt.  

The primary one is not having a direct contribution to the end-goal. Obviously old-salt engineers know a good build system will save thousands upon thousands of hours in the long run. Just ask anyone that had to deal with a make system where dependencies were defined properly and -j couldn't be used. Or had to work with a system where incremental re-builds didn't work. Or spend half a day debugging in the bowels of a legacy build system.

Another key attribute of this kind of problem is that it is deceptively complex. Similar to how everyone tries to re-invent logging frameworks and message queue frameworks only to discover that the perfect solution is a unicorn - a white whale. The deficits in the other implementations were actually trade-offs, not architectural flaws that some inferior engineers overlooked.

My own recent experience with this topic was learning YOCTO+bitbake, and maybe similarly build root. Working with embedded systems, I was so optimistic that finally a build system came along that would wrangle the workflow of source code -> image -> flash. Could something finally provide full bi-directional dependency mapping between and image and source code? Could I look at a binary in build output and instantly discover what source went into that file, what scripts/rules created, what decided to put it in that filesystem location in the output? No. It improved many things significantly, especially when it comes to configuration management for custom Linux, but in the end you have to learn an entirely new (arbitrarily different) scripting syntax just to end up with something that is essentially as complex and daunting as a custom make-based system. Never-mind trying to set something up from scratch - it would be an entire project in itself if you didn't have the base provided by poke or the hardware manufacturer.

Anyways, build systems are a fascinating meta-topic in software engineering. My guess is that in 50 years, most software will still be built by make. And petabytes of bandwidth and megawatts of electricity per day will still be used to send "HTTP/1.1 200 OK\r\n" in ASCII plaintext as if the computer on the other end is reading the response aloud to a human being. Another tale of technical debt born of legacy compatibility and brute force compensation.