Racket – Lisp beyond Clojure

A DSL may mean that the language doesn't fit how you think about a given problem, sure, but a poorly fitting abstraction can happen with or without those language abilities. I tend to think that having those poorly fitting abstractions expressed in a compact form is probably better than having them expressed in libraries with very large surface area and oddly connecting/dependent parts.

Now, I don't actually know this to be true; my experience with Lisp projects is very limited. I have worked with Perl and Tcl that made use of higher level capabilities of those languages to create little DSLs for specific tasks, and I didn't find it problematic; but neither is as far along as Lisp on the DSL spectrum. But, I have also worked with big projects in languages like PHP that get the abstractions very wrong (at least, wrong from my perspective), and have found the verbosity to be exhausting...saying so much in order to accomplish so little (and to do it wrongly) seems even worse than tiny amounts of code that don't fit my mental model.

In the general case, as always, "it depends". I'm of the opinion that shell scripts are very powerful for quickly prototyping something, but as soon as you need to do real software engineering (e.g. test suites, etc.) then it's probably worth porting over to a saner language (yes, there are test frameworks for bash; no, that doesn't make bash's semantics any more sane ;) )

Piping data between subprocesses is almost universally painful in anything other than a shell, so these sorts of libraries are great for that sort of glue; I wouldn't use them for anything that's not a bog-standard "foo | bar | baz | ..." though, since that's what the rest of the language is for!

In this case it was a no-brainer: the project requires a lot of s-expression manipulation, which is implemented as small racket scripts; these are called from bash scripts which contain the main logic, data flow, etc. I wrote it this way since I'd never used racket before, but figured it would be better suited to this s-expr manipulation than bash, python, haskell, etc. (which it is!); the remaining parts were simple enough to do with bash.

This worked fine for a while, but a recent change in requirements has invalidated a lot of the code's assumptions, so I need to ensure I'm making changes in the right place, that test cases are updated to reflect the changes, etc. which motivates porting to something more sophisticated, and racket is the clear choice here.

I wanted to use a shell-interop library after previously finding Scala's "process" package to be very pleasant to use. It's certainly made the porting job much more manageable, as it just becomes a case of recursively refactoring each part:

    bash:
    echo "foo" | ./bar.sh  | ./baz.sh  | grep "quux"

    racket:

    ---- Convert shell scripts to racket functions

    bash:
    echo "foo" | ./bar.rkt | ./baz.rkt | grep "quux"

    racket:
    (define (bar) ...)
    (define (baz) ...)

    ---- Move pipeline over to racket
    
    bash:

    racket:
    (define (bar) ...)
    (define (baz) ...)
    (run-pipeline '(echo "foo") '(./bar.rkt) '(./baz.rkt) '(grep "quux"))

    ---- Call racket functions directly, rather than invoking scripts

    bash:

    racket:
    (define (bar) ...)
    (define (baz) ...)
    (run-pipeline '(echo "foo") `(,bar) `(,baz) '(grep "quux"))

    ---- Encapsulate stdio

    bash:

    racket:
    (define (bar) ...)
    (define (baz) ...)
    (with-input-from-string
      (with-output-to-string (lambda ()
        (run-pipeline '(,bar) '(,baz) '(grep "quux")))
      "foo")

    ---- Replace stdio with arguments and return values

    bash:

    racket:
    (define (bar x) ...)
    (define (baz x) ...)
    (string-join (filter (lambda (line)
                           (string-contains? line "quux"))
                         (string-split (baz (bar "foo"))
                                       "\n")))

    ---- Replace strings with more useful datastructures

    bash:

    racket:
    (define (bar x) ...)
    (define (baz x) ...)
    (filter (lambda (line)
              (string-contains? line "quux"))
            (baz (bar "foo")))