Docker Compose best practices for dev and prod

One topic I'd like to see more discussion around is how best to set up staging environments for Docker Compose projects.

Ever since the V2 migration, I've been banging my head against multiple bugs in Docker Compose itself, which has severely damaged my trust in it as a production-ready utility. I no longer feel like I can safely update Compose without running extensive testing on each release.

For example, my journey since migrating to V2 has been:

* Originally on 1.29.2

  - Stable and I did not encounter any issues.

* Swapped to 2.3.3.

  - As part of the migration from V1, docker-compose lost track of several networks and started labeling them as external. Not a huge deal.

  - docker-compose stopped updating containers when the underlying image or configuration changed (#9291). This broke prod for a while until I noticed nothing was getting updated.

* Updated to 2.7.0, which was the latest release at the time.

  - This introduced a subtle bug with env_file parsing (#9636), which again broke prod in subtle ways that took a while to figure out.

* Updated to 2.8.0, which was the latest release at the time and before the warning was put up.

  * This broke a lot of things as it renamed containers and generated duplicate services, etc.

* Finally swapped to 2.9.0. No issues yet, but I'm keeping an eye out.

> For development, engineers want to run a local throwaway mysql/redis. In production, engineer want to use a proper managed mysql/redis.

Bad idea. You develop against a completely different local system, then you push to prod, and "oh no it doesn't work". Yeah - because you're developing against a completely different system! It behaves differently. It causes different bugs. You take a ton of time setting up both different systems differently, duplicating your effort, duplicating your bugs, having to fix things twice for no reason, and causing production issues when expectations from local dev don't match prod reality.

Every experienced systems engineer has been repeating this for decades. Listen to them. They all say exactly. the. same. thing. "Make all your systems as identical as possible." That doesn't mean make all your systems different whenever it is more convenient for you. It means that if it's possible for you to be developing using the same tech you use in prod, you should be doing that, unless it is impossible. If you think it's not possible or practical, you simply haven't thought hard enough about it.

The whole selling point of the cloud is that you can spin up environments and spin them down in minutes and pay pennies for it. Not a shared environment where everyone's work is constantly conflicting, but dedicated, ephemeral environments that actually match production. Literally the only example I know of where this doesn't work is with physical devices where you have a lab of 10 experimental pieces of test gear and you have to reserve them to test code on them. That can make remote development difficult. Still not impossible though.

> The cpu/memory requirements will be different.

Unless you're developing using the same servers as prod. There is no reason that you have to use your laptop. If you think "it's more convenient!", that's only because you have done zero effort to make Prod-like ephemeral environments more convenient.

> For development, engineers will build and run the docker image locally. In production, these are separate, you would build and publish the image separately in CI, and run only published images in prod. Again, makes sense.

That is the antithesis of what Docker was created for. The point of containers was to say "it works on my machine", and use that exact same image in production. Not a different image in CI that nobody developed their code against. It's just going to result in bugs that the developer didn't see locally, so they will need to spend more time to get it working. Your laptop can do the same build CI can on the same branch and push the same image that CI can.

What I'm saying is nothing new. Look into cloud-based development environments and the dozen different solutions made solely for rendering your local code in a remote K8s pod as soon as you write to a local file. Imagine a world where you don't waste your time testing something twice and fixing bugs twice.

im talking about managed tools... not self hosted ones.

e.g. datadog, etc https://www.datadoghq.com/blog/monitoring-kubernetes-with-da...

that said, Grafana's own hosted solution "Grafana Cloud" gives Kubernetes operators...but not for anything else. https://grafana.com/docs/grafana-cloud/kubernetes-monitoring...

the entire ecosystem is supporting k8s. there's not much choice here.

It's a great convention over configuration decision (that you can overwrite with -p) that trivially gives solution to multi-tenancy problem, similarly you'll run into problems when renaming $HOME to something else; for curl it's more like being surprised that it uses different source ip address and gives different result when called from host in China than in Europe.

If the dev environments are on AWS with real S3 buckets (etc), and everything runs as an auto scaling group, is it not possible to use the same exact docker images (and terraform) in dev and prod?

We haven't achieved it (yet?) but that's the direction I was aspiring to!