Live data from Hacker News

Why Databases Are Not for Docker Containers

myopsblog.wordpress.com

11–20 of 184 posts

Re: Why Databases Are Not for Docker Containers

#11
Part of the problem is that you're using databases that can't cope with failure. In large scale production systems things fail all the time. If you've got tech that can cope with failure it's not an issue.

Additionally, Docker is pretty handy when you're attempting to manage clusters consisting of thousands of nodes. In that instance enforcing best practices, automating workflows, scaling teams, auditing and preventing configuration drift are much bigger problems than a single server failing.

Re: Why Databases Are Not for Docker Containers

#12
post #6
post #3

Earlier quoted context omitted.

And when your journal is corrupted due to a bug in the Docker volume driver?

Then you let Docker write to a virtual network drive, streaming the journal out of the container.

I doubt thats going to be fast.

However, if you are going to be running docker on real tin (because thats where the value/speed comes in, if you're on AWS thats a whole 'nother issue) Then you might as well use device mapper for what it was originally designed for: mapping fibre channel. (or iscsi, or SAS [another scsi])

That is assuming you want speed, and have paid enough cash to overcome SPoF in your storage layer (it'll be cheaper and faster than trying to software your way out of it.)

Re: Why Databases Are Not for Docker Containers

#13
post #4

This article mentions offhand that the storage drivers are unreliable, even for data volumes. Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?

I can imagine that the probability of this happening is higher than when you run without docker. But on the other hand you have mechanisms like sharding and replications to deal with single machine/zone failure.

I have run database clusters on kubernetes in production without running into this particular problem.

The current state of container orchestrators for running databases is not optimal because one size does not fit all database types like with stateless applications. One solution for this problem are coreos operators which introduce third party resources into kubernetes that are specific to the database type and contain logic to manage this specific database type on kubernetes.

Re: Why Databases Are Not for Docker Containers

#14
post #9

I've been using docker in production with Elasticsearch and MySQL for 3 years in the PB scale and have never had data corruption issues occur. Corruption occurs on data drives even without docker - you still have to plan for it. This is why you enable replication. This is why you snapshot/backup your data daily and have disaster recovery plans. There are some major reasons why I actually think running databases in do…

Wow, three years ago? Balls of steel!

Re: Why Databases Are Not for Docker Containers

#16
> I’ve seen DBMS containers running on the same host with service layer containers. But these service layers are not compatible according to hardware requirements.

> Putting your database inside the container, you’re going to waste your project’s budget. Why? Because you’re putting a lot of extra resources to the single instance. And it’s going out of control. In cloud case you have to launch the instance with 64GB memory when you need a 34. In practice some of this resources will stay unused.

For some software, resource consumption is of "fixed" size, plus temporary workload-dependent growth (e.g. application-layer processes, most of the time.) Whereas some other software will take up all the space available to it (like DBMSes.) The latter are what resource quotas are for.

Containers are not meant to be treated like "Unix binaries but more easy to deploy." Containers are just lightweight VMs that don't have to do screwy things with memory balloon drivers to efficiently pack many of those "fixed plus temp growth" workloads onto a host.

But like VMs, containers still need resource quotas to ensure they don't thrash one-another. You can avoid specifying quotas for your fixed-with-temp-growth workloads, to "oversubscribe" a host, and it'll work (similarly to oversubscribing memory-ballooed VMs.) But the "all the space available" workloads need quotas.

The author might be used to public clouds, where VMs have a "size" in vCPUs + memory and that "size" is charged for, and so might not think of picking an instance size for a VM as explicitly setting a quota. But when you set up your own hypervisor cluster, you still have to decide how big each VM should be, regardless of the fact that a bigger VM doesn't "cost" anything: a VM's "size" is the compromise you make between the needs of that workload, and the ability to "fit" other workloads alongside it on a host.

But, to go further: if you're designing "instances" and running dedicated workloads on them, you're very likely "doing containers wrong." (This is probably a provocative statement; stay with me.)

Containers are to container hosts as VMs are to hypervisors: in both cases, their architecture assumes that if you want resource-efficient deployment, you've got a big generic cluster of hosts, and your guests are loaded onto them using a bin-packing algorithm (taking into account which guests need what extra resources that are only available on certain hosts, etc.)

If you don't have a big generic cluster of hosts, then your only packing options will be necessarily sub-optimal. If your container hosts are real hardware, you're out of luck; if your container hosts are themselves VMs, running on some cloud provider, then costs will be heavily in favor of taking advantage of the cloud-provider's bin-packing by wrapping each of your containers in a separate VM and then deploying those VMs.

(Which is, coincidentally, what Amazon's Elastic Beanstalk does for you, and why it's not the same as Amazon ECS. ECS is for setting up your own "big generic cluster" of container hosts to bin-pack across; Elastic Beanstalk is for wrapping containers in VMs so that AWS will bin-pack at their abstraction level.)

Re: Why Databases Are Not for Docker Containers

#17
I still can't really figure out what a container is. Every time I think of a use case for one, I read something like this which says that's a terrible idea.

The use-case I need solved most often is the following:

Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the other "servers" I have running. Do I just want a bunch of VMs? Or docker instances that all point to some other DB (that's apparently not in a docker instance...?). But then they're no longer independent from one another because they all use the same DB. So do I need a separate DB for each serverlet? Which lives where? On its own VM?

Re: Why Databases Are Not for Docker Containers

#18
post #5

Earlier quoted context omitted.

Isn't it possible to stream a copy of the database out of the container and over the network to another database instance?

Chicken and egg. Is that other instance containerized?

Decoupled failure modes, unless you think there's a global "fail all docker data stores".

Re: Why Databases Are Not for Docker Containers

#19
post #17

I still can't really figure out what a container is. Every time I think of a use case for one, I read something like this which says that's a terrible idea. The use-case I need solved most often is the following: Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the…

There is nothing special about containers to really understand

Containers are a lightweight way of sandboxing a process. Think a level lower than a VM. You can run multiple containers on a single VM in the same way you can run multiple VMs on a single host.

Ideally a container should be stateless. If a container crashes, you should be able to bring it up again without anything actually caring that it is technically a different process.

A container doesn't solve a "real problem" it mostly makes it easier to manage applications and processes by abstracting out any dependencies from the host VM and keeping everything packaged into a single thing.

A container can run any application that it is configured to run on any VM regardless of the state of the VM (Assuming the VM has a kernel that supports containers)

Re: Why Databases Are Not for Docker Containers

#20
post #17

I still can't really figure out what a container is. Every time I think of a use case for one, I read something like this which says that's a terrible idea. The use-case I need solved most often is the following: Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the…

> I still can't really figure out what a container is.

If you mean this in a general sense...

One use case at Unbounce (where I worked in infrastructure) was to encapsulate the runtime dependencies for different services that were on a machine.

Our monolith required Ruby 2.1 and a bunch of gems. Then we were using Scout for centralized monitoring, which required 1.8 with a separate set of gems. We only noticed the problem when our monolith moved from Ruby 1.8 to 2.1.

To fix this problem of dual-Ruby runtimes, we encapsulated the Ruby 1.8 + gems into a Docker image for Scout, then ran the Scout container on the machine. It works perfectly and never conflicts with the monolith's Ruby runtime.

Post reply on HN