Understanding the Power of Lisp (2020)

When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.

In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.

[1] https://www.youtube.com/watch?v=mSgXWpvQEHE

1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form.

2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros.

3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1]

The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk.

[0]: https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async.

[1]: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.

This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.

[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.

[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.

  for f in GetFromCache, GetFromDatabase, GetFromAPI, Compute:
    try:
      return f()
    except Exception as ex:
      log(ex)
  return default_value