Ask HN: Who operates at scale without containers?

We use Docker for registry and code delivery. We use Kubernetes for rollouts. These are not critical, we periodically reevaluate if we want to continue using them. We've designed our software to work well even if some components shut down, so we don't need any fancy rollout strategies: just take everything down and boot a new version.

Our strategy:

1. Go codebase. No CGo. Build artefacts are static binaries. They are copied into Docker containers. 2. That's all.

Build a base AMI using Packer and launch it to an Auto Scaling Group behind a load balancer. Deploy code to the ASG using CodeDeploy. Use RDS for the database.

This is a good match for languages that have good concurrency like Elixir. They benefit from deploying to big machines that have a lot of CPU cores, and keeping a common in-memory cache on the EC2 instance is more efficient than using an external cache like Elasticache. It also works well for resource-hungry systems with poor concurrency like Ruby on Rails. Putting these kinds of apps into big containers is just a waste of money.

Here is a complete example of that architecture using Terraform: https://github.com/cogini/multi-env-deploy

Similarly, bare metal can be really cost-effective. For $115/month, I can get a dedicated server with 24 VCPU cores (2x Intel Hexa-Core Xeon E5-2620 CPU), 64 GB RAM, 4x8 TB SATA, 30 TB traffic (see https://www.leaseweb.com/dedicated-servers#NL). That would be an order of magnitude more expensive on AWS with containers.

For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores?

Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s).

Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology