Rolling your own servers with Kubernetes

A lot of areas of expertise do go away though. I don't need to worry about the electrical features of the building where I'm hosting servers. No worries about whether I set my automated backups for the prod DB right because RDS makes it trivial. No worries about physical security and having to be anal about site access management or physical segregation between hardware and users. And so on and so forth.

Physical server management is in general a liability and cost center.

> Ops, security, and reliability don’t go away when using EC2 either, which is what led the massive cloud explosion.

I'd argue that in many cases they do get significantly easier in a cloud environment especially for lightweight setups.

For example, if you run a managed sql database in aws backups are largely handled for you. Upgrades take a handful of clicks and ~5 mins. You dont have to think about patching the underlying operating system.

Broadly speaking, cloud providers give you lots of tools to make things that were fiddly much easier.

As an ex big iron commercial UNIX sysadmin, let me tell you; they can do it way better than most sysadmins.

Developing IT policies for physical security, server security, update policies, purchasing, wiring and so on takes a huge amount of knowledge, and the possibility that a middle-of-the-road sysadmin is not only competent at, but excels, in all those areas is literally zero.

Cloud providers have world-class experts in each area whose sole responsibility is providing for their specific area. It's economies of scale at its best. It doesn't mean that it's always the best choice, but having to have someone do full-time server maintenance at no additional value to yourself or customers is just a huge drag.

Some thoughts:

1) Do you really need to invest a million dollars in bin-packing containerized stateless microservices?

2) You don't have to use K8s to get the benefits of rolling your own servers. In fact, I'd argue you should do the latter well before you do the former.

3) Definitely hire someone who has done it before. You will save so much time and money your head will spin.

4) Do not just build a rack full of random commodity gear. Make sure it is suited specifically for your purposes, and then weigh the cost of service contracts and managed colo against a $100k+ cage monkey on call 24/7.

5) Do not fall for the "We've got , we don't need redundant hardware!" lie. The more parts of your system rely on lots of hardware, the more fragile your system becomes. Distributed decentralized services become a PITA when the underlying gear is flaky, and centralized services require it. Do not underestimate the shittiness of your colo; always design for the most redundancy you can get for what you have. If you can run dual power, do it. Dual network stacks, do it. Redundant disks, do it. Remote management, do it. Outside modem to a management port on the router, do it. Always be postponing entropy. Later, when you become a FAANG (yeah right) you'll have the time and money to automate away some of the redundancy issues.

6) It's sometimes harder to upgrade disk or bandwidth on an existing machine than it is to just buy another machine, but the more machines you have, the more problems (and overhead costs), and there is an upward limit on the scaling of most services without rebuilding them. So buy big on local disk and network, and buy new machines to expand cpu and memory. The redundancy and performance issues inherent to SAN/NAS often make local disks a better choice, as they are unlikely to impact your whole network at once, you can fix/upgrade them piecemeal, and they are less difficult to manage/operate.

7) Don't DIY critical components such as data replication; it's surprisingly hard. In fact, don't even believe vendors if they claim they can solve X for you with some magic software or hardware. Get them to show you a live demo on your own network.

8) Don't forget that in 3 years you'll be replacing it all.

Back then you basically had to do it, so everyone did. These days you have to compete with companies that make use of the flexible offerings of AWS/GoogleCloud/Azure. Which for small companies is a large advantage, since it makes it possible to start up with almost no money, and scale up smoothly, while letting your employees be focused on the product you offer for your customers.