Dockerizing MySQL at Uber Engineering

Mostly what I was commenting on was how they are building some type of declarative infrastructure where they define the topology they want and then system will build it. This was screaming k8s to me.

However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it might make sense to fragment massively.

For example, lets say database #235 is using 75% of a shared server. Then at least in the very short term migrate the numerous other databases at the container (or image) level to other servers so the overloaded database doesn't flood out every other business system you support.

Now the argument is if you ran mysql bare metal then you'd have more memory and maybe only use 25% of that shared server, but sooner or later you'll get a big enough flood of traffic that you'll want to segment off and your operations team understands docker at 2am but not mysql DBA stuff quite as much. Sometimes its nice to have everything use the same standard and everything lives in docker might work.

This also has fascinating forensic and QoS implications where you can do interesting snapshot and cloning tricks affecting precisely and exactly one DB at a time.

The original motivation was to support multiple hardware generations. PCIe flash cards have become much larger over the years very quickly -- much faster than older cards become end-of-life. The result is that if you're running a large fleet of database hosts for many years, their storage capacity will differ greatly, both between datacenters (e.g. older DCs will have older hardware on average) and eventually within a datacenter that gets a partial refresh.

By defining automation configuration that is smart enough to know that some hosts get 1 mysqld and others get 2 mysqld, based on storage / hw generation, there's a much better flash utilization win.

This setup also enables faster replacement of failed hosts. Say each host has N mysqld, all part of different pools. If a host fails then you need to hot-copy the data set of each of these, from other replicas in each affected pool, to a new location. The trick is the replacements can copy from N different source hosts, and even go to N different destination hosts as well. This permits massively faster hot copying behavior vs having a single giant mysqld per host.

tl;dr it requires a lot of automation but there are very valid reasons for doing this.

That said, I would not advocate using Docker to achieve this. It provides little benefit for this scenario. If you're good at calculating mysqld memory usage, you can already just set the buffer pool and per-session buffers to a size that prevents multiple mysqlds from ever swapping. Meanwhile cpu and network rarely are points of saturation for a db host so that tends to work out fine without a quota system.

So that leaves i/o as the main resource that the processes will compete for. But Docker cannot provide i/o isolation.

From what I understand, Google/YouTube containerizes their data stores, but their systems around containerization are far more advanced than anyone else's. So I assume they've already solved this problem internally, but that doesn't mean the current state of the art in the open source world is up to the task yet.