What scripts have you built to stand up a new server?

For one startup, the MVP included deploying some Linux-based overseas factory stations, under initially impossible conditions.

For reasons that were really good in context, the software aspects of our station builds were based on a set of step-by-step written manual instructions, to configure the Linux distro to "bootstrapping" state for running the huge configuration shell script.

The huge shell script was written to be idempotent. So it could do both the initial heavy configuration and then the same or later versions of it re-run to make any engineering-change adjustments. Including re-run safely remotely, after the machine was deployed on the other side of the globe, in an active production line. If you're comfortable with Bash or Python, a script like this can be easier than shoehorning the problem into the structure of some declarative tool.

(The really good reasons would sound crazy out of context, and you wouldn't think it worked, but we did ultimately have perfect software uptime. And, yes, there were plans (before the Covid chill on our customers and VC) to do a lot of things differently in our next-gen station, including putting most of it in a container.)

Besides bespoke scripts (Bash, Python, Make), I've also used declarative off-the-shelf tools that do a lot of heavy lifting, including doing things with Terraform. (For example, Terraform to allocate a set of AWS EC2, storage, and networking, to simulate a real metal server environment, for developing some infrastructure software.) But, even with those tools, there's always some kind of documentation of it, and often also bespoke scripts.

Regarding remembering all the steps and getting them right... One of the tricks to documentation for things like "pets" servers is to have a canonical internal wiki into which pretty much all useful info that's not otherwise in Git goes. For example, if you go to the AWS Console to add or change something, have the right wiki page open, and update it, including copy&pastes, as you go. (Also, cross-link that with any issue-tracking for the task.) If you keep wiki access low-friction and high-value, it should save you at least as much time as it costs, and be even more valuable to others. (Put loosely, I've seen situations in which a project or a person's job hinges on whether or not one key sentence was captured in a wiki.)

Sometimes, this documentation can later be turned into a script.

Following these lightweight doc conventions, even my personal laptop has a wiki page on how to reconstruct its configuration, and it's kept up-to-date for years.

This is probably an unpopular opinion, but I have a bunch of install scripts that install some programs from source. Even some basic things like vim. The reason being that there's some customization I want to do. A lot is that I often can't trust the package manger. For simple things like idk if I'll get python3 (or even python) support in vim to the fact that Ubuntu 20 had fd-find and batcat while Ubuntu 22 has fd and bat. The other side of this is that I don't always have full control over the machine so I'll just install things into `"${HOME%/}"/.local/{bin,builds,include,lib,share}`. There's rarely "one-size-fits-all" solutions, but if we know what decisions we will want to make, we can leverage that.

This does also end up having a multiplying effect where if I want to use ansible I can call these scripts directly (which I find often easier than using ansible itself...). Then usually I can have the system set itself up, or at least 90% of the way and while it doesn't save wall time, it saves my time. I can also make specific options for the distro at hand or when I have certain constraints (it is easy to probe for quickly and that can be put in a common file that can be sourced by other scripts). The scripting method is helpful because you're just doing the same design pattern as when programming: containerizing functions, creating flexibility, modularity, and readability (I also highly suggest putting notes in these scripts. Not for others, for you. The more you automate the quicker you'll forget the awesome tricks you found, but you'll be more likely to remember where to revisit). Because like you said "it's very repetitive." When you see that, then you know there's a great opportunity to leverage your programming skills.

I purposefully try to make these scripts require minimal tooling. My main ones are `curl`, `grep`, and `sed`. So I can generally rely on having those on a fresh system. This is really all you need (though you should use these to grab things like `make`).

Pro tip: make a template maker. I have one for a github source. While I can normally just source my common file, there is some benefit for having everything self contained. You won't be able to write the whole script this way but you definitely can get all the boiler plate out of the way which is at least 80% of it, and hey, maybe you could get an LLM to do another 10% for you. Though it I haven't found one that's really that good at bash. (I suspect that this is primarily due to the metric ton of shit bash scripts, since the average one is beyond terrible). Idk why bash scripting is a "lost art" but it is not that hard (for the love of god, use functions).

I also suggest writing some systemd and cron skeletons. These can save a lot of time and really help as you find mistakes or if you want to add extra system hardening. I do find that common implementations do not have the containerization I want (since you're mentioning hardening). You can't always trust the ArchWiki, but it is usually mostly right. An example might be with Fail2Ban, where I don't like to put logs in /var/log/fail2ban/ instead of /var/log/fail2ban.log{,.{1..N}}

I'd share but I don't want to dox myself here but I'm happy to share some bash tips or other quick hacks.

(to be clear, not everything is or should be installed from source. You don't have infinite time and I don't used Gentoo).

You don't use cloudinit with Docker containers. You'd use cloudinit to set up the Docker/K8s host on EC2 or similar IaaS.

Software for Docker containers is built like this: https://12factor.net/

> A lot is that I often can't trust the package manger.

If you can't trust the package manager, how can you trust the install media?

Strongly do NOT recommend what we are doing

- https://github.com/chrisgo/fabric-example

- https://github.com/chrisgo/fabric2-example