Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker Compose Isn't Enough

blog.tealok.tech

Post reply on HN
Print view View original

81–90 of 208 posts

Re: Docker Compose Isn't Enough

#81

Post by hamandcheese »

chefandy wrote: post #27
zemo wrote: post #20

people dramatically overestimate how difficult it is to write a program that controls docker for you. This is one of those things where you can write like two pages of Python and ignore... all this: > Tealok is a runtime we’re building for running containers. If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.

I'm not dev-ops-familiar with Docker, so you might be more familiar with the problem space, but it seems like "You can just write a Python script to do what you want" is the sort of thing people say to justify not giving people the config infrastructure they need to solve their problems without having to write executable code. Like, sure, you often can make up for not having a zoom lens by walking towards what you're…

> to solve their problems without having to write executable code.

Some people would rather write 1000 lines of yaml than 100 lines of python, and I really don't understand why.

Re: Docker Compose Isn't Enough

#82

Post by forrestthewoods »

theshrike79 wrote: post #76
forrestthewoods wrote: post #53

Earlier quoted context omitted.

Many abstractions are very bad. Stacking bad abstractions on bad abstractions is why modern software is so slow, laggy, and bloated.

What would be your solution to running, say, 15 different Python applications on the same machine, each requiring a unique set of library versions?

Create 15 pex files. Or possibly 15 PyInstaller executables. Then simply run them like normal programs.

Re: Docker Compose Isn't Enough

#83

Post by vander_elst »

ricw wrote: post #28
Izkata wrote: post #8

I've always understood docker-compose to be a development or personal tool, not for production. Is that not usually the case? Also aside, "docker compose" (V2) is different from "docker-compose" (V1) [0], it was rewritten and integrated into docker as a plugin. It should still be able to handle most old compose files, but there were some changes. [0] https://docs.docker.com/compose/releases/migrate/

it's perfectly fine if everything can run on a single server instance. which is probably the majority of things. i've run production instances with 2PB of data being scraped per month by 100+ CPUs and 256GB of RAM using docker compose. some of the machines (smaller instances) have run flawlessly with zero reboots for years on end. both on cloud and on-prem. i'd say its 100% production ready.

+1 to this. Smaller datapoint from my side I guess, but anyway, docker is the core of my self hosting setup and it is one of the things that I don't have to fiddle with, it just works.

Re: Docker Compose Isn't Enough

#84

Post by number6 »

cdrini wrote: post #24
ricw wrote: post #14

So the problem is 1) port mapping and 2) backing up data volumes? There are simple solutions to each 1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for deta…

For 1, docker has a handy special file to make this even easier! `compose.override.yaml` loads automatically on top of the main `compose.yaml`. We keep dev configs in there so Devs only need to run `docker compose up` locally without any environment variables, but on prod (which is automated) we set the compose file to be `compose.yaml:compose.prod.yaml`.

I do it the other way round, so in production you just have to use `docker compose up`. I did not understand the `compose.yaml:compose.prod.yaml` syntax or never saw it. What does this do?

Re: Docker Compose Isn't Enough

#85

Post by number6 »

shubhamjain wrote: post #18
ricw wrote: post #14

So the problem is 1) port mapping and 2) backing up data volumes? There are simple solutions to each 1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for deta…

Yup, I was surprised how weak the arguments were. I mean, surely there are better reasons why docker compose fails to scale? I have been pretty happy with it right now, and haven't felt the need to try something like k8s or docker swarm.

Sounds more like a skill issue. I am quite glad with traeffik and docker-compose. But I don't have that high loads or interactions on my servers

Re: Docker Compose Isn't Enough

#86

Post by cdrini »

number6 wrote: post #84
cdrini wrote: post #24

Earlier quoted context omitted.

For 1, docker has a handy special file to make this even easier! `compose.override.yaml` loads automatically on top of the main `compose.yaml`. We keep dev configs in there so Devs only need to run `docker compose up` locally without any environment variables, but on prod (which is automated) we set the compose file to be `compose.yaml:compose.prod.yaml`.

I do it the other way round, so in production you just have to use `docker compose up`. I did not understand the `compose.yaml:compose.prod.yaml` syntax or never saw it. What does this do?

Oh if you set the environment variable COMPOSE_FILE, you can specify a colon-separated list of compose files, which get merged together. This lets us have our core services in our main compose.yaml, since they're shared between dev and production, and then have extra services that are prod-only, or toggle certain prod-only environment variables, inside the compose.prod.yaml . And extra services/settings which we only want in the dev environment go in compose.override.yaml .

Eg

    COMPOSE_FILE="compose.yaml:compose.prod.yaml" docker compose up
My work is OS so you can check it out here if you wish: https://github.com/internetarchive/openlibrary/

Re: Docker Compose Isn't Enough

#87

Post by shiroiushi »

forrestthewoods wrote: post #75
shiroiushi wrote: post #66

Earlier quoted context omitted.

Containers are a convenient work-around for the problem where programs have incompatible dependencies, and additionally the problem where security isn't as good as it should be. For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only co…

> For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful. This is because the Linux model of global system wide shared dependencies is stupid, bad, and wrong. Docker and friends are a rou…

The Linux model works fine (very well, in fact, because of less HD space and much more importantly, less memory used for shared libraries) for programs that are normally included in the Linux distribution, since the whole thing is built together by the same organization as a cohesive whole. If every random little 20kB utility program were packaged with all its dependencies, the bloat would be massive.

It doesn't work very well for 3rd-party software distributed separately from the OS distro and installed by end-users.

The problem I've seen is that, while pre-Docker there was really nothing preventing ISVs from packaging their own versions of dependencies, they still only targeted specific Linux distros and versions, because they still had dependencies on things included in that distro, instead of just packaging their own. The big thing is probably glibc.

As I recall, Windows went through a lot of similar problems, and had to go to great lengths to deal with it.

Re: Docker Compose Isn't Enough

#88

Post by justsomehnguy »

ricw wrote: post #28
Izkata wrote: post #8

I've always understood docker-compose to be a development or personal tool, not for production. Is that not usually the case? Also aside, "docker compose" (V2) is different from "docker-compose" (V1) [0], it was rewritten and integrated into docker as a plugin. It should still be able to handle most old compose files, but there were some changes. [0] https://docs.docker.com/compose/releases/migrate/

it's perfectly fine if everything can run on a single server instance. which is probably the majority of things. i've run production instances with 2PB of data being scraped per month by 100+ CPUs and 256GB of RAM using docker compose. some of the machines (smaller instances) have run flawlessly with zero reboots for years on end. both on cloud and on-prem. i'd say its 100% production ready.

And I've seen compose services not surviving a banal host restart and require manual massaging.

So it's 100% production ready... for you.

Re: Docker Compose Isn't Enough

#89

Post by oliwarner »

ricw wrote: post #14

So the problem is 1) port mapping and 2) backing up data volumes? There are simple solutions to each 1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for deta…

> So the problem is…

This feels like quite a shallow reading. Being able to manage system-wide things (DNS, and a centralised Caddy, centralised database) from individual compose units in such a way that you're not going through a ton of setup each service. This much might just need Caddy to work better with other instances of itself but it isn't a solved problem.

I'll grant you that backing up volumes isn't hard.

Re: Docker Compose Isn't Enough

#90

Post by eliribble »

denkmoon wrote: post #65
wmf wrote: post #9

Earlier quoted context omitted.

It's called content marketing. IMO this article is content marketing done right since it has useful information even if you don't read to the end.

Looks like a grift. Note the copyright on the Tealok page, then have a gander; https://gleipnirinc.com/investors

Blog post author here - that's the right company name, but not the right company website. Our company doesn't have a website yet.

Looks like we may share a company name with some grifters. That's what we get for making our name an obscure mythological reference.

Post reply on HN