Live data from Hacker News

Why Databases Are Not for Docker Containers

myopsblog.wordpress.com

171–180 of 184 posts

Re: Why Databases Are Not for Docker Containers

#171

I am so surprised and disappointed that such a shallow article has made it to the top. It provides absolutely no value. Most of the upvotes (gathering a consensus from comments) are not because they believe that Docker is not the right tool but because they have been frustrated by the ops part of things. I'm also surprised at how many think that one tool will come and solve all their problems. Guys, it doesn't work t…

There are little to no rant against apt, yet there are a lot about Docker.

Trends always have an explanation...

Re: Why Databases Are Not for Docker Containers

#172
Hi everyone. I'm this immature who's made this blog post. Thanks for the feedback, I didn't expect so much people to care about my post, cause I mostly lead the blog for myself. There were about 20000 users seeing this while I had only 3000 for the last year. So, if you have any questions for me as an author - please, write them down at this thread. I'll try to deal with all comments soon.

Re: Why Databases Are Not for Docker Containers

#173
post #130
post #25

> But what about Configuration Management systems? They’re designed to solve this kind of routine by running one command. The problem with this for most of the developers you see praising containers, is that with a containerized setup, you've already got the rest of your deployment process down to `docker service update --image myorg/myservice:1.3.0 myservice` (And, in fact, maybe you're even running that code agains…

This sounds right to me. As a DBA containers appear to be a nightmare. I'm employed as an absolute expert in my product. A developer may know how to use Docker but are they an expert? Now... * Who is going to look after the middle ground when the database is in the container? * Who is going to be responsible for rewriting enterprise tools to discover those instances to gather metrics? Because none of the traditional…

Most of your arguments comes down to "this doesn't fit in my world where windows is king, so it won't work for me". That however, is not a problem of containers.

While I don't consider myself to be a pure DBA, I do know Postgres quite well, and manage quite a few both "classic" deploys in a VM and containerized instances. I was the one who created a default Postgres setup/image/config that our devs use, which when it's used correctly and as documented, when it is deployed to production, it is exactly the same as managing a normal instance.

For the devs it's simple, their local env is a checkout of a sample env, copy that to their new project, docker-compose up, and they have a database running with pretty much the same config they would get in test, acceptance and production. No surprises, we both know what to expect.

Backups? Still the same. Patches? I tell my config management to a pull a new postgres image on the servers and restart the db images during a maintenance window. This makes it actually a lot easier than updating the non-containerized services.

> and the business suffers a massive loss of data

This scenario should be recoverable in the first place, and should be tested on regular basis. I'm actually setting up a process to automatically verify database recovery using containers, which makes stuff like this a lot easier and more convenient. Spin up container, restore backup to it, full vacuum analyze, pg_check, select counts from every table, select random records from every table, and if possible spin up a test instance of the application (again, very easy if that also runs in a container) where we can run unit tests against the restored database.

> who are they going to sue? A liability-free open source project?

So when have you last heard about someone suing MS or Oracle when they had data loss? I suggest you read your license agreements... Our entire business runs on such "liability-free" open-source projects. Linux, Postgres, GNU userland, Python, GCC, clang, Boost, Wildfly, Java, ... and it worked out pretty well for us. We're not some hipster startup with nodejs, angular and mongodb "cloud" apps, we provide some mission-critical services for clients that are banks, oil companies, governments, ... with corresponding SLA's. The attitude of our (very tech-focused) management is simple: we don't need liability umbrella's when we _own_ the technology and know what the hell we're doing. If something does go wrong, this would mean that yes, we would be responsible, no point in hiding.

Re: Why Databases Are Not for Docker Containers

#174
post #25

> But what about Configuration Management systems? They’re designed to solve this kind of routine by running one command. The problem with this for most of the developers you see praising containers, is that with a containerized setup, you've already got the rest of your deployment process down to `docker service update --image myorg/myservice:1.3.0 myservice` (And, in fact, maybe you're even running that code agains…

This is why I appreciate DBaaS offerings... It makes more sense to run DBs at least closer to the hardware, outside containers, but most developers don't want to be DBAs, so it's better to pay for someone that has all the maint/update scripts written, and not have to deal with many of those issues.

If you run database containers on a bare metal docker/container os (coreos/rancher/...), your DB's will probably be running a lot closer to the hardware than they would in a VM.

Re: Why Databases Are Not for Docker Containers

#175
post #62

Earlier quoted context omitted.

> Containers are a lightweight way of sandboxing a process. Think a level lower than a VM. can you go into a little more depth? my understanding of a VM is that it installs the OS in a dedicated memory partition, and allocates hardware resources separately from that of the host machine, such that resource contention between host and VM never happens. the VM allocated resources just go dark for the host machine while…

old way: one chroot per user and quota (including ulimit) per user new way: limits handled with cgroups security with namespace related to user's profile. namespace can do syscall limitations, and also offer unique network access per users. Roughly simplified. freebsd jails where known as glass jails that would eventually break, LXC/docker are the same except they hav'nt broke yet at the price of more complexity. The…

> They will all dramatically break when people will have figured the trick in 5 years.

VM's hypervisors have also seen bugs/exploits to gain hypervisor access, and yet they're still here. These things have been fixed, and people did not stop using them.

Also, I don't really see the "more complexity and obfuscation" thrown at "the problem", can you enlighten me?

Re: Why Databases Are Not for Docker Containers

#176
post #157
post #144

Earlier quoted context omitted.

It's kind of interesting how knowledge works with some subjects. If you don't know anything, you will agree with the statement. When you know a bit more, you will disagree, but when you learn more than that you will once agree. It's true that real databases need to guarantee data won't disappear even at power loss, so you would think that container crash should be comparable with power loss, if not more trivial. The…

You stopped one iteration short. You assume you can, and will, know that your OS and your disks do not lie to you. Yes, persisting data in a consistent and durable manner is hard. It is damn hard. It was hard 20 years ago when systems required to store obscene amounts of data started to become more common and it is hard today. (This reminds me of a discussion a couple of years ago on how to kill processes. There are…

Yes it does lie especially with fsync() because that call's purpose was to flush all caches to disk, which is an expensive operation.

Since then NCQ/TCQ were added to disks and also systems like Linux implemented write barriers to enable more control[1].

[1] https://monolight.cc/2011/06/barriers-caches-filesystems/

Re: Why Databases Are Not for Docker Containers

#177
post #163

Earlier quoted context omitted.

>> 500+ large scale production deployments This needs to be qualified.... Did you deploy a single system 500 times, or 500 different systems? Or some combination thereof.

It's a mix. I'm a consultant that specializes in large scale distributed systems. I have some customers that have >100k production database nodes. I manage probably >50PB of data. I have designed large distributed systems for more than 100 customers.

Nice. 100K database nodes! Is that like, Facebook or Twitter?

I hope you write about that somewhere.

Re: Why Databases Are Not for Docker Containers

#178
post #152

Earlier quoted context omitted.

Google runs MySQL on Borg internally.

Same for Bigtable and Spanner.

I assume these guys have their own network controller and kickass optic fiber links. Network attached storage in poor cloud environments leads to issues.

Re: Why Databases Are Not for Docker Containers

#180
post #62

Earlier quoted context omitted.

old way: one chroot per user and quota (including ulimit) per user new way: limits handled with cgroups security with namespace related to user's profile. namespace can do syscall limitations, and also offer unique network access per users. Roughly simplified. freebsd jails where known as glass jails that would eventually break, LXC/docker are the same except they hav'nt broke yet at the price of more complexity. The…

> They will all dramatically break when people will have figured the trick in 5 years. VM's hypervisors have also seen bugs/exploits to gain hypervisor access, and yet they're still here. These things have been fixed, and people did not stop using them. Also, I don't really see the "more complexity and obfuscation" thrown at "the problem", can you enlighten me?

Old ways: you trust a person. And You check.

New ways: you create very complex sets of syscalls permissions that can be fine tuned by using a Role Based Accounting or any Auth/Profile Framework.

The idea is that you got rid of the risk of mischief/compromission by containing the code and people by delegating trust to external stuff : companies, datacenters, external servers, an OS you don't own. But for delegating you use software to delegate en masse :)

However, now, your surface of attack is so big that it's impossible to do a full audit of your perimeter. And people focus on code/practices/network. You have delegated a lot.

For the sake of discussion what can be the next cost efficient approach for attack with that much smoke?

The downside of containers is the physical geographical increase of the perimeter to defend, and for SV to develop their beautiful code, a lot of workers (cleaning personal, transporter, electricians, construction workers, firemen) that are required in the physical world and are so impoverished that they are becoming a vulnerability.

Keep It simple, always attack where the costs are less.

Bribing a man today to access a physically a server, a router is less expensive than writing an exploit.

The obvious problem in the containers is the idea you can trust layers you should not. Maybe, your container runs in a datacenter where a worker infected a printer with a connected cam from home because he is to poor to afford a printer? Maybe from there you can compromise a router, and have a MITM on a VLAN used between 2 servers?

Who knows? But how can you know if you cannot check?

Post reply on HN