Live data from Hacker News

Why Databases Are Not for Docker Containers

myopsblog.wordpress.com

91–100 of 184 posts

Re: Why Databases Are Not for Docker Containers

#91
post #79
post #4

This article mentions offhand that the storage drivers are unreliable, even for data volumes. Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?

In order for a database to provide such guarantees it needs some guarantees from an underlying hardware. Mainly that when disk say that data was written, it really was written, and that the data was written in the right order. The danger with overlay systems is that they might not provide these guarantees, which makes database writes unsafe. Given that current overlay drivers are unstable as it is, I doubt they put a…

> Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM

This is true for any and all file systems. ext4 can't provide those guarantees either because it's up to the hypervisor to do the right thing when the guest requests it to flush caches to disk.

ZFS just happens to have made it very explicit that it requires certain guarantees to provide the data reliability, but those same requirements exist for other file systems.

Re: Why Databases Are Not for Docker Containers

#92
There is actually little difference between a process running in a container and in the host. They are using the same network and ideally the same filesystem so there should be no difference whether you run the database or app in the host or the container.

Every container article on HN seems to perpetuate more confusion on containers and often arbitrary misguded rules on what a container should be confusing new users even more.

The problem is Docker has taken fundamental technologies developed by other people and wrapped it and since they do seem to want to give credit and pretend to be more than what they are they obfuscate things in words like docker filesystem drivers, networking drivers, container drivers etc. Untill users get familiar with the underlying technologies this sorry state of affairs will persist.

A container is simply a process launched in its own namespace thanks to kernel namespaces introduced in Linux 2.6. Its got nothing to do with cgroups, cgroups 'can' be used to limit resources to container process by cpu, memory or network resources if you want. If you launch this process by chrooting (or pivot root) into a basic linux rootfs filesystem you have a container. If you launch an init in this process you have a LXC container. If you don't and prefer to launch the process directly from the host you have the Docker version of the LXC container which is a fussy hack as now your container is not contained and runs app process not designed to be run in pid 1 in pid 1 and needs to be managed from the host. Kudos. You can also add a network namespace to the container process so its has its own network layer.

The biggest problem currently is a lot of Linux subsystems are not namespace aware and you can't really do proper isolation. Even cgroups only recently got namespace support. Anyone know who these folks are who are doing all this fundamental work?

The second biggest problem is layers are oversold, their actual practical use is marginal at best. They are also complicated and buggy with multiple issues with running overlayfs or aufs on xfs, databases and btrfs. The third biggest problem is a lot of projects and teams working on Linux containers are pushed into the background or marginalized and misrepresented like LXC was by Docker devs instead of giving proper credit and explanations.

The talented developers of overlayfs, aufs for instance are virtual unknowns in the container ecosystem inspite of Docker fundamentally dependent on them. These guys can solve a a lot of the problems with containers but first users must know about them and support them so that bugs can be fixed, rather than have the Docker team create more workarounds and hacks.

Re: Why Databases Are Not for Docker Containers

#94
post #54

This article is terrible. It's a lot of wishy-washy explanations devoid of technical detail - because their isn't a technical explanation or justification for this list. I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster). Any clustered…

Hadoop and HBase are very different from mysql. Those run on yarn and are designed for containers.

> Any regular relational database should be pretty close to automated failover

In my experience, most people who work with mysql would not enable automated failover. And I believe the concerns in the article are valid and important if considering a container for mysql.

Edit: Though I do think it conflates containers and things like kubernetes or mesos in an awkward way. The good arguments are more about running relational dbs in containers on some sort of cluster orchestration system.

Re: Why Databases Are Not for Docker Containers

#95
post #54

This article is terrible. It's a lot of wishy-washy explanations devoid of technical detail - because their isn't a technical explanation or justification for this list. I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster). Any clustered…

While I agree with you I'd like to caution some users about rushing into dockerising everything in their production environment. If your environment setup is not repeatable and you don't have your configuration management under control then you have other problems and using docker is just going to add another layer of abstraction on your mess that your DBA doesn't know how to deal with when things hit the fan. In particular I can imagine improper understanding of docker volumes could bite some people, but they also have some questionable defaults for networking (user land proxy, rewriting iptables)

That being said we currently use docker for some of our production databases, mainly for almost-idle services (mongodb for graylog, zookeeper for kafka), but I have had no problem using them for some moderately sized services with a couple thousands writes per second on redis/kafka (which is nothing for them).

We're still using non-containerised versions of the databases that needs dedicated bare metal servers mostly because I don't see the risk-benefit being worth it, but I'd love to hear someones war stories about running larger scale databases in docker.

For development, I don't think there's anything better for databases, it beats manual setup, vagrant boxes, and shared development servers by a long shot. I feel that educating everyone on your team in how to use it is well worth the investment. docker-compose makes setting up even a fairly complicated development environment a breeze.

Re: Why Databases Are Not for Docker Containers

#96
post #78

Earlier quoted context omitted.

Are you telling me that you'd trust MySQL auto replication failover enough to have it activates multiple times per day (even with Percona)? On a busy cluster with, say, 300GB?

Seems like a straw-man. Why is a process running within a container failing over more frequently than a process running directly on bare metal -- seems like more of a resource/process scheduling issue than anything to do with containers.

Leaving aside there's now yet another abstraction layer to have bugs, it is not a straw-man to imply that current container technology is not as reliable as bare metal.

Or do you mean that Docker etc. are as reliable as bare metal?

Heck, you can go on Docker's webpage (https://www.docker.com/what-docker). Notice that while they make the claims that Docker is-

1. More lightweight

2. 'Open'/run anywhere

3. Secure

They don't say 'more reliable'.

I use Docker for systems I design; I also recognize current container technologies have their limitations. It is my job to know and avoid these pitfalls.

Re: Why Databases Are Not for Docker Containers

#97
post #78

Earlier quoted context omitted.

Seems like a straw-man. Why is a process running within a container failing over more frequently than a process running directly on bare metal -- seems like more of a resource/process scheduling issue than anything to do with containers.

Leaving aside there's now yet another abstraction layer to have bugs, it is not a straw-man to imply that current container technology is not as reliable as bare metal. Or do you mean that Docker etc. are as reliable as bare metal? Heck, you can go on Docker's webpage ( https://www.docker.com/what-docker ). Notice that while they make the claims that Docker is- 1. More lightweight 2. 'Open'/run anywhere 3. Secure The…

This conversation should distinguish between Docker as a product and container technology in general.

A lot of issues that people encounter with Docker specifically disappear if you run Kubernetes (such as volume management), simply by ignoring what Docker does and doing something sane instead.

> Or do you mean that Docker etc. are as reliable as bare metal?

This doesn't really mean anything, Docker the product has a lot issues - sure. Container technology in general? No. Where do you draw the line here? Is a chroot 'as reliable as bare metal'? At what point is a container not running on bare metal anymore?

Re: Why Databases Are Not for Docker Containers

#98
post #54

This article is terrible. It's a lot of wishy-washy explanations devoid of technical detail - because their isn't a technical explanation or justification for this list. I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster). Any clustered…

Yeah volumes skip unionfs. This article is full of FUD. The author demonstrates they don't really have enough experience to make these claims. I wonder if google has database nodes in containers? Kubernetes is adding the features for containers now. I think it is stable now.

> I wonder if google has database nodes in containers?

Yes

Re: Why Databases Are Not for Docker Containers

#99
post #43

Alright, all you brave people who run databases in docker: where do you store the actual data? - Host mounts? So what happens when a container gets rescheduled to another node? - Docker volumes? What happens when a container gets rescheduled to another node? - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess? - Shared filesystem like NFS, or Ceph? How's the performance fo…

I use ZFS under everything (including docker) and so I can use scheduled snapshots, backups, etc, to do the real data management, and not worry about losing data.

I can use host mounts too, but I prefer to just back up and manage all of docker, out of band, without thinking too much about it.

EDIT: granted this is for relatively small databases though. I think it could be scaled up however.

Re: Why Databases Are Not for Docker Containers

#100
post #43

Alright, all you brave people who run databases in docker: where do you store the actual data? - Host mounts? So what happens when a container gets rescheduled to another node? - Docker volumes? What happens when a container gets rescheduled to another node? - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess? - Shared filesystem like NFS, or Ceph? How's the performance fo…

Mesos with placement constraints solves this problem. In case of AWS, store the data on an EBS volume, use constraints to dedicate agents with specific hardware to the DB containers. Always ensure that EBS is available to the agent on which the DB supposed to run. Easy. Same for Kafka, same for Cassandra, you name it...
Post reply on HN