What scripts have you built to stand up a new server?

I first run a bunch of checks to try and make sure the build will be successful. Then create the instance where cloud-init does a few basics to allow the rest to work. Once the instance is up, the connection in Ansible flips from localhost to the server, mounts the drives, installs everything that needs to be installed and does whatever configuration is needed, and adds records in whatever systems of record need to be updated. The whole process takes about 10 minutes or so (for a single sever), depending on some external dependencies. The time increases as the count goes up. That’s probably something we could solve for, but it hasn’t been a big deal so far.

Up to you, but just in you weren't aware: ansible has a `raw:` task which enables one to run anything that the `connection:` transport tolerates, which can go a long way toward getting rid of any extraneous shell scripting

Further toward that goal, I found that PyPy is "untar and go" for every distribution that I've cared about, so even getting python on the machine can similarly be bootstrap friendly

you know that it won't drift. you can generate images from the same config you apply to a host. you can stamp out the same config across N hosts. you can completely reconfigure a host without reprovisioning it from scratch. you can run slim VMs by mounting the nix store as read only.

it would be great to run an entire company off nixos this way

> Idk why bash scripting is a "lost art"

I suspect that has to do with the fact that writing or reading it is often an effort in futility, acting as an efficient rabbet hole time trap.

Bash scripting is supposed to act as glue, and it does so extremely poorly. It relies on program writers coding their interfaces correctly to maintain determinism (which many have done very poorly). It makes no guarantees, or warnings about such errors, it makes it quite easy to make these mistakes, and the respective maintainers of the various utilities it may depend on have said these bugs are working as intended, nofix.

Take a look at ldd sometime, I'm sure you'll notice the output has three different types of determinism problems which prevent passing the output to any automation (and having it work thereafter correctly), without first patching the software.

That particular bug was reported in 2016, it was partially fixed in 2018 by PaX, but the maintainers wouldn't pull the fix (from what I read), so PaX forked it. The bugs still exist there today afaik.

For glue to work, you have to be able to make certain guarantees and be able to correctly transform the output in various ways easily. Visibility in all processes is extremely important as well and changes made on one version should continue to work on later versions.

Unfortunately, by externalizing most of the basic functionality to various core utilities, you potentially get different behavior every time you update, and as mentioned it fails resiliency tests; instead becoming brittle. There is also very little visibility without having deep knowledge of the ecosystem.

Ironically, what was described in the Monad Manifesto got this better than anything else I've seen since.