Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker Compose best practices for dev and prod

prod.releasehub.com

Post reply on HN
Print view View original

71–80 of 166 posts

Re: Docker Compose best practices for dev and prod

#71

Post by dijksterhuis »

remram wrote: post #58
dijksterhuis wrote: post #57

Earlier quoted context omitted.

I have lost count of the times I've heard "but it works fine on MY machine, it must be the CI/CD that's broken" I then find out that no-one bothered to run the compose file that matches what gets put into prod on their local machines. My point is: your view might become more black and white on this matter after the N-th "but it works fine on MY machine" comment. EDIT: Where N = your personal tolerance level of bullsh…

Sounds like this was easily caught by CI, which does run the prod compose file, and everything was fine.

This wasn't a real example. Just a generic thing that happens everywhere I go.

There's always one in the crowd that refuses to give up local development.

Re: Docker Compose best practices for dev and prod

#72

Post by nickjj »

JadoJodo wrote: post #62

One of the things we've really struggled with is sharing dev compose files between Windows and macOS. We were unable to make bind mounts work correctly with Windows, due to performance issues (this used to be documented in the Docker docs, but I can't seem to locate it now). This resulted in us falling back to VSCode's remote container tooling. Has anyone else had this issue with Windows? The last time I looked into…

> We were unable to make bind mounts work correctly with Windows, due to performance issues (this used to be documented in the Docker docs, but I can't seem to locate it now)

Docker Desktop volume performance on Windows is great if you're using WSL 2 as long as your source code is in the WSL 2 file system. It also works great with WSL 1. I've been using it for a long time for full time dev.

In fact, volume performance on Windows tends to be near native Linux speeds. It's macOS where volume speeds are really slow (even on a new M1 using Virtiofs). Some of our test suites run in 30 seconds on Windows but 3 minutes on macOS.

But the Docker Compose config is the same in both. If you want, I currently maintain example apps for Flask, Rails, Django, Node and Phoenix at https://github.com/nickjj?tab=repositories&q=docker-*-exampl..., I use some of these exact example apps at work and for contract work. The same files are used on Windows, macOS and Linux.

The only time issues arise is when developers on macOS forget that macOS' file system is case insensitive where as Linux is not. Docker volumes take on properties of their host so this sometimes ends up being a "but it works on my machine!" issue specific to macOS. CI running in Linux always catches this tho.

Re: Docker Compose best practices for dev and prod

#73

Post by pbalau »

_joel wrote: post #59
bdcravens wrote: post #18

Surprised it didn't mention profiles, which let you launch only part of your "stack" (think background processing in addition to web app, etc) https://docs.docker.com/compose/profiles/ Prior to that feature, I had a hand-rolled approach using bash scripts and using overrides. Worked well enough, but felt like a dirty dependency.

Neat, this would have saved some headaches in the past (about 2018). Was it a feature then? If so I wasted some time reinventing the wheel, it seems!

It's quite new, I'm still waiting for some free time to ruin it.

Re: Docker Compose best practices for dev and prod

#74

Post by kossae »

CincinnatiMan wrote: post #15

I recently learned that moving the compose file around and running commands on it with it in different directories puts things into a bad state. This is because it apparently tags images and containers with context on where the compose file was run from. However, you can negate this by always passing in a custom project name with -p.

To mitigate this, I usually always require a .env file for docker-compose containing directories with at minimum a `COMPOSE_PROJECT_NAME` envvar to prevent these naming issues. As long as the env file travels with the directory there are no issues.

Re: Docker Compose best practices for dev and prod

#75

Post by bdcravens »

_joel wrote: post #59
bdcravens wrote: post #18

Surprised it didn't mention profiles, which let you launch only part of your "stack" (think background processing in addition to web app, etc) https://docs.docker.com/compose/profiles/ Prior to that feature, I had a hand-rolled approach using bash scripts and using overrides. Worked well enough, but felt like a dirty dependency.

Neat, this would have saved some headaches in the past (about 2018). Was it a feature then? If so I wasted some time reinventing the wheel, it seems!

It was added in early 2021 (1.28)

Re: Docker Compose best practices for dev and prod

#76

Post by dijksterhuis »

gottie wrote: post #21

I find docker compose override files whilst powerful ultimately much more confusing and error prone to work with. Instead I would recommend just duplicating your compose files and changing each separately, perhaps using a templating engine. It is often much easier to work with a single compose file containing all the config in one easy to read block. This compared to having to do a yaml merge of all the various block…

I usually suggest passing `--env-file` to the actual compose binary. Gives you the "multiple deployment versions" without faffing about with external templating tools.

    docker compose --env-file ${INSTANCE}.env -f docker-compose.yml up
In the `*.env` file:

    CONF_A_VALUE="some_value_a"
    CONF_B_VALUE="some_value_b"
    CONF_C_VALUE="some_value_c"
    ...
In the compose file:

    services:

      service-1:

        # you need to be more explicit here
        environment:
          SOME_NAME_FOR_A: "${CONF_A_VALUE:?err}"
          SOME_NAME_FOR_B: "${CONF_B_VALUE:?err}"

        # but it gives you simple templating in other places
        labels:
          com.company.name.label.a: ${CONF_D_VALUE:?err}
          com.company.name.label.b: ${CONF_E_VALUE:?err}
          com.company.name.label.c: ${CONF_F_VALUE:?err}
        ports:
          - "${CONF_G_VALUE:?err}:8020"

      service-2:

        # also gives services more freedom to diverge from one another (if required)
        environment:
          MAYBE_ANOTHER_NAME_FOR_A_BECAUSE_COWBOY_TEAM_REASONS: "${CONF_A_VALUE:?err}"
          SOME_NAME_FOR_C: "${CONF_C_VALUE:?err}"
Where `?err` forces the variable to be set and non-empty: https://docs.docker.com/compose/environment-variables/#subst...

It's just a shame that `docker stack` doesn't have an `--env-file` argument yet for swarm deploys.

Re: Docker Compose best practices for dev and prod

#77

Post by pcorsaro »

edfletcher_t137 wrote: post #39

The article is titled "for dev and prod" (emphasis on the latter), yet the very first suggestion is to bind-mount your source outside the container so you can mess with it at any time? How is this good advice for production?! (Update: this is indeed great advice for development, and it's actually in a sub-section dedicated to that! My bad.) Update: reading comprehension, it's a thing! :) Thanks chrsig for pointing ou…

Just curious, why would a bind-mount be bad in production? I understand if you're running multiple app servers and need the code to be deployed in a lot of places that they'd be bad, but if you're just using a single server, what's the downside of using bind-mounts? You could use git to push code changes up to your server, and they'd be picked up immediately without having to rebuild and push an image.

Re: Docker Compose best practices for dev and prod

#78

Post by bluedino »

tmpz22 wrote: post #69
JadoJodo wrote: post #62

One of the things we've really struggled with is sharing dev compose files between Windows and macOS. We were unable to make bind mounts work correctly with Windows, due to performance issues (this used to be documented in the Docker docs, but I can't seem to locate it now). This resulted in us falling back to VSCode's remote container tooling. Has anyone else had this issue with Windows? The last time I looked into…

You're trying to create a uniform developer experience across two platforms that are wildly different. If you find yourself spending a ton of time on a one-size-fits-all-disappoints-everybody solution, maybe its time to build two effective solutions instead.

That's kind of the point if containers, isn't it?

Re: Docker Compose best practices for dev and prod

#79

Post by tmpz22 »

bluedino wrote: post #78
tmpz22 wrote: post #69

Earlier quoted context omitted.

You're trying to create a uniform developer experience across two platforms that are wildly different. If you find yourself spending a ton of time on a one-size-fits-all-disappoints-everybody solution, maybe its time to build two effective solutions instead.

That's kind of the point if containers, isn't it?

Oh god no, Linux containers were not intended to be the solution for cross compatibility issues between Windows and MacOS

Re: Docker Compose best practices for dev and prod

#80

Post by twblalock »

0xbadcafebee wrote: post #30

If you don't use the same Docker Compose file for Production, don't use it for Development. Your two different systems will diverge in behavior, leading you to troubleshoot two separate sets of problems, and testing being unreliable, defeating the whole "it just runs everywhere" premise. Docker Compose is fine for running smoke tests/unit tests, but if Dev and Prod run differently, there's really no point to using Do…

Also, don't use Docker Compose in dev if you use Kubernetes in prod. Just use the same thing everywhere. Otherwise the fact that something works in dev tells you nothing about whether it would work in prod.
Post reply on HN