Dockerizing MySQL at Uber Engineering

> Running multiple instances of a DB server means that each DB is going to be hard-coded to a limit which is some fraction of the available memory.

What you are suggesting then, is setting limits in a way that does not guarantee resources for any given database.

> There are limits even to this, though. The most memory you can get for any single, broadly available instance in AWS is 244 GiB, and its going to cost you an arm and a leg to run. It's not hard to create a dataset which exceeds 244 GiB in size.

Most people don't have databases that large. If you do, perhaps you shouldn't be puting multiple of them in a single instance if you're limited to machines that small.

For comparison, I have servers with dozens of databases that fit into memory on machines with 64GB and less. I also have machines with single databases that need far more. If we need to, we'll provision machines with 1TB-2TB of RAM.

> This isn't something I frequently see being an actual problem, within the same corporation.

It's something I see all the time. Consider that "within the same corporation", people often run databases for a large number of external customers, or need to be able to roll out new versions, or do development and testing on different versions.

I have at least 5 different versions of Postgress sitting on production servers right now due to customers with different requirements and different upgrade cycles. I also have several different versions of MySQL.

Bare metal is operationally very expensive because someone has to monitor it, back it up, patch it, test it, fix the software when it breaks at 2am, fix the hardware when it breaks at 2am, explain to security that its either full of confidential stuff or not, decommission it years later, and none of that is as standardized as virtualized or containerized stuff.

Also "in the olden days" we had prod servers but shared dev and test servers. Now a days you just spin up resources assuming your ops is flexible enough. I'll spin up a test image to eliminate one bug and then destroy it, no sweat. In theory you can do that with metal but the accounting must be weird. You could make a pool of bare metal test boxes for people to use as they please, I guess...

I also like spinning up new images for software upgrades. Oh, a new version of the database, here's a new image, test it out.

Some of it is organizational hacking. After a few legendary disasters procedures will be formulated where standing up new iron takes interdepartmental meetings and signing off with the network and security and ops teams and the data center guy has to sign off on the thermal and electrical loads and power points all over the place. In comparison, you wouldn't make someone changing a cell in a spreadsheet go thru all that, right? So you deploying a virtual image is just clicking a harmless little button, as long as you operate under a blanket agreement with ops, infosec, networking, etc... At least until enough legendary disasters inevitably happen that clicking "create" on a virtual image requires weeks of time and at least 4 signatures and 3 departmental meetings of micromanagement. Hopefully we'll invent something new by then.

Why? There are tons of advantages of using docker that's not just for running multiple processes on a single machine. You can still gain advantages from just installing a single database container on a big server + docker volumes.

Being able to version control image creation scripts, being able to pull a identical image for both dev/production is useful and quicker than trying to use puppet on dev machines, the fact you can now use your docker deployment infrastructure for all services, and not have exceptions for things.

Whats the disadvantage? No one has explained this to me.

> As a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea.

Storage capacity is also growing exponentially: Samsung is shipping a 15TB SSD (albeit at $10000), Seagate has previewed a 60TB SSD.

> You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required.

The limits of RDBMs are well understood. You don't need any fancy hardware for this use case. A couple of Xeons, as much RAM as you can afford and a RAID of SSDs (or possible spinning disks, it doesn't look like Uber is doing anything too fancy).

SPOF: you have slave replicas running that can take over if something goes wrong with the master.

You don't have to take down an RDBM to do maintenance. DDL statements are transactional in PostgreSQL.

The automatic sharding of Cassandra is nice, when you need it. Of course, Uber's use case seems like it lends itself to easy geographic sharding when you're using an RDBM, if needed.

In the end, I'd rather deal with a mature well-understood technology like an RDBM compared to a 5 year old technology like Cassandra (release 1.0 in 2011). You obviously prefer the opposite. To each its own.

> does not guarantee resources for any given database

Correct. Instead it gives the most resources to the pages in all of the databases which are accessed and used the most. MySQL and InnoDB are remarkably well tuned to ensure that the most often used pages are in memory, where they can be accessed and updated with the greatest performance.

> It's something I see all the time.

We apparently work on very different usecases. Fair enough. Just be aware that your usecase is far from typical. Continuing to argue points across such diverse usecases isn't going to make for a productive discussion.