The Ultimate Frontend Build tool: make

The problem I seem to be having with various dotfiles tools is they seem to be doing too much, where a script with a bunch of calls to ln -s would do.

There us Premake which is Lua version and can be configured to do anything you want, and will run on anything.

I've come to think that build systems are a very personal utility. Everyone has their favourite. Mine's fabricate.py, for example. Over time I've built up a library of script snippets and shortcuts and so on which I'm familiar with, comfortable with, and exactly fulfil all my use cases. It's all very clever. :)

Yet when I download some random project's source code, I groan at any sophistry in the build process at all. I'm not interested in your build system - I'm interested in the application itself. Maybe I want to try and fix a bug, or have a half-baked idea for a new feature. I don't need dependency checking, incremental rebuilding, parallel building, and all that stuff you get from a fully operational build system at this point. I only need to build the project - once - as I decide whether to stick around. Sure, if I start working on it for serious, rebuilding over and over - then I'll bother to learn the native build system, and read any complicated scripts. Build systems are an optimization for active developers. They're a utility that is supposed to save time.

Of course, you're never going to get everyone in the world to agree on the same build system. We all have different desires and needs for what machines it should run on, how automated, how much general system administration it should wrap up, how abstractly the build should be described, etc. It's a bit like one's dot files or choice of text editor - my ideal build is tailored just for me but I wouldn't expect it to satisfy anyone else.

So now I wish that everyone who distributes software as source code would do this: include a shell script that builds the project. Just the list of commands, in the order that they are executed, that carries out a full build on the author's system. That's what it comes down to, in the end, isn't it? Your fancy build system should be able to log this out automatically. (Of course then you still include all the fancy build stuff as well, for those interested.)

Of course it's extremely unlikely that your shell script will work on my system without modification. There's probably machine-specific pathnames in there for a start. We might not even use the same shell! It's basically pseudocode. But if I'm faced with a straight list of imperative shell commands that doesn't work, and a program of some sort with its own idiosyncratic syntax and logic and a hundred-page manual and the requirement for me to install something - which also doesn't "just work" - well, as long as you know how to call your compilers and linkers and so on - which you should - the former is going to be easier to tweak into submission, to get that first successful build. After all, if I need much more than that I'll probably just recreate the build in my favourite system anyway.

I've (ab)used apenwarr's redo for a couple big data processing projects, with mixed results.

One was a news recommendation engine. We pulled down and parsed RSS feeds, crawled every new link they referred to, crawled thumbnails for each page, identified and scraped out textual content from pages, ran search indexing on the content, ran NLP analysis, added them to a document corpus, ran classifiers and statistical models, etc.

Every step of the way took some input files and produced an output file. We used programs written in many different languages -- whatever was best for the job.

So a build system was the obvious way to structure all of this, and we needed a build system we could push pretty hard. Our first version used make and quickly ran into some limitations (essentially, we needed more control over the dependency graph than was possible with the static, declarative approach) so we turned to redo, which lets you write build scripts in the language of your choice.

One thing we needed almost immediately was more powerful pattern matching rules than make's % expansion. No problem: invent a pattern syntax and a special mode where every .do script simply announces what patterns it can handle. Collect patterns, iteratively match against what's actually in the filesystem, and then you've got the list of target files you can build. (This already differs from make, which wants you to either specify the targets explicitly up front as "goals," or enumerate their dependencies via a $(shell ...) expansion and then string transform them into a list of targets which are ALSO matched by some pattern rule somewhere...okay you get it, it's make, it's really disgusting.)

Another thing we needed was to say, here's a list of target files that appear in the dependency graph, give me them in topologically sorted order. This allowed us to "compact" datasets as they became fragmented, without disturbing things downstream from them in the dependency graph. Again, this was not difficult with redo once we had some basic infrastructure.

Now, was all of this maintainable, or was it just kind of insane? I think in the end it ended up somewhat insane, and most importantly, it was an unfamiliar kind of insane. The insanity that you encounter in traditional Makefiles is at least well understood. And treatable.

With redo, you can do almost anything with your build. You can sail the seven seas of your dependency graph. It's awesome. It's also terrifying, because there is very little to guide you, and you may very well be in uncharted waters.

But give it a shot anyway. YMMV.

Me too.

I use make together with a markdown compiler, and the m4 preprocessor, to keep devdocs up to date, in one huge document, where everything is included, and the various sections as stand-alone docs. The markdown version of the section files is almost uncluttered from m4 and html. I link from any external doc to any other via links in the toc in the main doc index.html, to keep everything as simple as possible. It's sweet.