ClojureScript Builds, Rebooted

It sounds like Boot has had a lot of thought put into it, and I certainly welcome the idea of effectively immutable directories.

But it also looks like Boot dives head first into I/O, when I'd prefer a build tool that is a little more circumspect about complexity. While I welcome competition to Leiningen, and I'll certainly keep an eye on Boot, my initial impression is that it's heading in the opposite direction to where I'd want a build tool to go.

I'm of the mind that build tools aren't going to get better unless we forcibly insert instrumentation between our build tasks and the stateful resources upon which they depend. To that end, I'd like to hear more about both your Clojure runtime isolation and filesystem isolation mechanisms.

There's certainly a lot you can do to constrain I/O, but removing side-effects when possible will always be better than merely restricting them.

I do like the idea of immutable files and repeatable builds, but I don't think this negates the benefits of maximising the time you spend working with data.

For instance, there might be a task that automatically cleans up some files, but you want to keep those files around. If the tasks produce data structures that describe the build process, it's much easier to intercept and prevent or reorder the cleanup process. Functions, especially ones that are Turing-complete, are notoriously opaque.

> removing side-effects when possible will always be better than merely restricting them

I disagree. Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns.

Furthermore, if you can intercept effects, you can impose purity upon them. For an extreme example, consider application virtualization and containers such as Docker. By intercepting the system call table, you can create a "pure" filesystem from the view outside the container. At the other extreme, take a look at "extensible effects" and the Eff language, which lets you stub any subset of the effects available down to the individual expression!

> If the tasks produce data structures that describe the build process, it's much easier to intercept and prevent or reorder the cleanup process.

If you intercept all file IO, you can recover the same data. The only difference is whether or not you know that data upfront.

> Functions, especially ones that are Turing-complete, are notoriously opaque.

This is true! However, there are a great many build processes that do not know what they depend on or what they will produce until they do some Turing-equivalent work. For example, scanning a C header to find #include statements.

Rather than try to shoehorn all data in to a declarative model, we need both 1) fully declarative and 2) the ability to recover a declaration from the trace of an imperative.

An example of this trick, employed manually, is the notorious .d Makefiles. The C compiler finds all the dependencies, produces a submake file with the .d extension, then make-restarts recursively using the new .d file as part of the dependency graph. However, it's a very unnatural way to think about the problem and it leads to complex multi-pass build processes that are necessarily slower. Instead, the dependency graph could be produced as a side-effect of simply doing the compilation and that graph could be used as part of a higher-level declarative framework.

Yes, in the post I didn't get too technical with the treatment of tasks; I'll elaborate a little here. First, the command line thing.

You're right that the command line isn't strictly required to use boot, you can do everything at the REPL or perl/awk etc., as you pointed out. But for me it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds because they can be very concise. Just like I probably wouldn't be super excited to use a Clojure shell instead of Bash, because Lisp is usually more verbose for the kinds of things I do on the command line. Consider:

    $ boot cljs -usO none

vs

    (boot (cljs :unified true :source-map true :optimizations :none))

the command line version is just nicer for that. When it's time to automate is when you'd put that in your build.boot and make it a new task.

This brings us to tasks. We used to describe them as "middleware factories" but Rich has provided us with a way cooler name: stateful transducers. The build process can be imagined as a transducer stack applied to a file set instead of to an async channel or sequential thing. The principal value the task abstraction provides is their process-building power.

A typical task definition looks like this:

    (deftask foo
      "This task does foo."
      [...]                  ; kwargs/cli-opts
      (let [state ...]       ; local state
        (fn [continue]       ; middleware
          (fn [event]        ; handler
            ...              ; build something, do work
            (continue event) ; call continuation
            ))))

Please ignore "event", it's there for historical reasons. But like transducers, we have powerful ways to build processes from tasks that don't need to know anything about each other now. For example:

    (deftask bar
      "This task does bar."
      [...]
      (comp
        (fn [continue]
          (fn [event]
            ...
            (continue event)))
        (foo :bar "baz" :baf "quux")
        (omg :hello "world")))

A key property of transducers is that they can also perform process control flow duties. The boot `watch` task, for instance, is a totally general-purpose way to to incremental-anything in boot. The `cljs` task doesn't have a file watcher in it, none of the other tasks do. They don't need it.

Another example is the `cljs-repl` task, which emits ClojureScript code when you start the CLJS REPL. This requires recompiling the JS file and reloading the client. This all happens automatically because the cljs-repl task can call its continuation whenever it likes, so it does that when you start the REPL. This means that your webapp code doesn't contain any REPL connecting code, so you don't have to think about removing it for production builds etc. The REPL connecting code is in there when you use the task, and not when you don't. Very clean.

Another interesting property of tasks is that they accept only keyword arguments. They do not take positional parameters. This means that partial application of tasks is idempotent, and that last-setting wins. For instance, given a function f that takes no positional parameters, we have:

    (-> f (partial :foo "bar")) ==
    (-> f (partial :foo "bar") (partial :foo "bar"))

and

    (-> f (partial :foo "bar")) ==
    (-> f (partial :foo "baz") (partial :foo "bar")).

This is pretty interesting because it gives us a nice way to manage global preferences. We have a macro called task-options! which can be used to globally apply options to tasks:

    (task-options!
      foo [:bar "baz"
           :baf "quux"]
      omg [:hello "world"])

This macro actually does some currying and alter-var-root, replacing the value of the task var (deftask defines a var, of course) with a curried version. A cool thing about this is that the last-setting-wins property means that you can override these settings on the command line or in the REPL:

    (boot (foo :bar "not-baz") (omg))

which would override the :bar option, but not the others.

This is probably long enough, hahaha! I'll hand the mic back to you now :)