Simple Is Not Small

I already preferred to keep source code files under about 200 LOC, so.

Brooks, in “the Mythical Man-Month”:

“One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams.

Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.

In the upper left of Fig. 1.1 is a program. It is complete in itself, ready to be run by the author on the system on which it was developed. That is the thing commonly produced in garages, and that is the object the individual programmer uses in estimating productivity.

There are two ways a program can be converted into a more useful, but more costly, object. These two ways are represented by the boundaries in the diagram.

Moving down across the horizontal boundary, a program becomes a programming product. This is a program that can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data. […] As a rule of thumb, I estimate that a programming product costs at least three times as much as a debugged program with the same function.

Moving across the vertical boundary, a program becomes a component in a programming system. This is a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks. To become a programming system component, a program must be written so that every input and output conforms in syntax and semantics with precisely defined interfaces. The program must also be designed so that it uses only a prescribed budget of resources — memory space, input-output devices, computer time. Finally, the program must be tested with other system components, in all expected combinations. […] A programming system component costs at least three times as much as a stand-alone program of the same function. The cost may be greater if the system has many components.

In the lower right-hand corner of Fig. 1.1 stands the programming systems product. This differs from the simple program in all of the above ways. It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts.”

The Unix pipeline and the similar-looking Clojure expression are entirely different in how they do what they do. Pipes are process abstractions. The (perniciously improperly understood) "pipeline" macro is not pipelining anything in any way (not CPU nor process nor memory). It is merely syntax sugar to write a deeply nested call chain as a "flat" list of operations.

And it goes on to compare "ugly" code, again making the category error that "more" code is "worse". Because, one is swapping / splicing entire programs within a pipeline. Also, there are plenty of ways to slice that mango; you don't need temp files.

That said, having intermediate files in one's pipeline is a big help because one can use those to make pipelines idempotent. Plus, one doesn't need flat files, one could swap in a SQLite cache too, at will, without modifying anything else in the pipeline. This kind of design change is not possible with the equivalent Clojure function call chain, as-is. In fact, having such a requirement (one always finds need to restart processes after crashes, and have them pick up from where they died) causes us to write some custom (and therefore design-wise brittle) conditional restart loop on top of the computation to manage its failures inside the running program. An idempotent Unix pipeline simply needs to be... restarted from outside the process.

In this particular case, a valid complexity complaint would be about the lacunae of the Bash / shell programming language (and interpreter model) versus the Clojure language.

Also, again from a program design point of view, I feel there's a bit of "holding it wrong" going on there... Bash / Shell-fu, yes, but not enough to be dangerous (for example, sort | uniq | sort is a standard idiom of pipeline programming). Which claim is personal, and so I'm open to being corrected at the same level. Sources: my code and writing:

https://www.evalapply.org/tags/bash/

https://www.evalapply.org/tags/clojure/

https://github.com/adityaathalye (the pinned repos are Bash and Clojure)

And specifically, this log processing code, for a more apples-to-apples comparison with OP's post.

https://github.com/adityaathalye/bash-toolkit/blob/master/lo...

  deduplicate() {}

  frequencies() {}

  drop_first_n() {}

  drop_last_n() {}

  drop_header_footer() {
     drop_first_n "${1}" |
         drop_last_n "${2}"
  }

  window_from_to_lines() {}