Wow, great post! I have the same complaints:
(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!
(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.
We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:
(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.
(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.
Debugging distributed systems locally could be the norm, but it's not.
The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).
(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.
XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:
https://news.ycombinator.com/item?id=25343716
Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.
Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.