Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#51

Post by toomuchtodo »

UK-AL wrote: post #38
memracom wrote: post #27

Earlier quoted context omitted.

If you would use something like Ansible it would be just as fast because it is fully automated, and there would be no work at all to change a config because Ansible will also customize config files from templates. Chef and Puppet and SaltStack work similarly, just learn how to use your tool of choice.

And for a lot of people the tools of choice is docker...

Picking the wrong tool because its easier never really scales.

Docker is not configuration management. Configuration management is not containerization.

Disclaimer: DevOps by day. I care because it makes it easier for me (and cheaper for you) to come in and scale things if it was done right in the beginning (instead of me having to revamp your infrastructure to move components to the right tools), but either way I get paid, so it doesn't matter much to me. I'm just trying to help make your life easier.

Re: Dockerizing MySQL at Uber Engineering

#52

Post by otterley »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

Additionally, the article appears to falsely assume that one can start a new container with a different and arbitrary MySQL version and reuse the data volumes from the previous container. From experience I know that this is not always the case: upgrades might work, but downgrades often will not. Even in the case of upgrades, various DB schema changes will be needed.

Re: Dockerizing MySQL at Uber Engineering

#53

Post by amenod »

> The MySQL data directory is mounted from the host file system, which means that Docker introduces no write overhead

If I understand correctly, Docker doesn't introduce any write overhead, mounted from host or not. This is the main difference from VMs - syscalls are just being "forwarded" to kernel. Or am I missing something?

Still, I am only arguing with the stated reason. Of course you don't want to save state inside an (ephemeral) container.

Re: Dockerizing MySQL at Uber Engineering

#54

Post by otterley »

memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

I wouldn't go so far as to say they should "never be containerized," but doing so doesn't solve any of your problems other than a packaging problem, and it arguably creates problems you might not have had before.

It's my experience that most of the impetus behind Docker is driven by developers who need it on their Mac/Windows workstations to run Linux processes, and who (for good reason) want to ensure that their development environment is as similar to the production environment as possible. In other words, the developers are driving the production environment, instead of the production engineers driving the development environment. This leads to friction when the production environment has its own unique set of constraints and needs that Docker and containers aren't quite yet a match for.

Re: Dockerizing MySQL at Uber Engineering

#55

Post by exelius »

VLM wrote: post #45

Most of the assumptions in the comments boil down to its a bad idea if you're resource constrained because you'll get much better performance when everythings on one big server with shared ram. However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it…

Exactly this.

You don't go to containerization because it's cheaper / more efficient, you go to it because it's more flexible and lets you simplify your tech ops into business ops faster. Containers require a LOT of infrastructure / overhead to build -- and don't be seduced by cloud providers that abstract that complexity, if you build a business on containers you WILL have to deal with it. But it does allow you to de-skill a lot of tech ops processes, which helps reliability (so long as your strategies are effective).

And you're right; depending on the way they handle their sharding (which may be part of a greater data availability strategy) this might not be so bad. But business goals are driving this, so you have to find a way to make the tech deliver what the business wants.

Re: Dockerizing MySQL at Uber Engineering

#56

Post by otterley »

amenod wrote: post #53

> The MySQL data directory is mounted from the host file system, which means that Docker introduces no write overhead If I understand correctly, Docker doesn't introduce any write overhead, mounted from host or not. This is the main difference from VMs - syscalls are just being "forwarded" to kernel. Or am I missing something? Still, I am only arguing with the stated reason. Of course you don't want to save state ins…

There's not even a forwarding process. Containers are simply processes running in a set of one or more different "namespaces." A mount point in a container is just an ordinary mount point from the kernel's perspective. In this case, the mount point is relative to the container's root filesystem set via chroot(2).

Re: Dockerizing MySQL at Uber Engineering

#57

Post by falcolas »

cwyers wrote: post #46
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

> 1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire. Really? The…

If you're storing the `mysql` DB part of your docker container, you're in for a bad time as well. There's a tremendous amount of mutating metadata which is stored in that DB which needs to be monitored.

Much better to store it as its own version controlled SQL file which is run against a newly started DB (that third command I referenced). You could break out every statement into its own provisioner command, but then you are beginning to muddle the line between machine-level configuration and database level configuration.

So yes, I still stand by my original assertion that you can stand up a fully functional DB with three Ansible/puppet/chef commands.

> maybe Uber has problems at a scale that you don't have

I will admit that it is entirely possible. But from what they present to the public (which is all any of us outside of Uber have to go on), their problems aren't really that unique: a heavy write workload which is easily sharded. Only their chosen method of resolving them is proving to be unique, and they appear to disregard any and all normal methods of resolving their issues.

Sometimes the boring way of doing things - accepting that critical stateful processes are indeed different from stateless processes and managing them accordingly - is still the best way. Getting away from this means getting away from MySQL and its ilk entirely. I hope we can get there one day; managing DB instances separately from your stateless instances is a pain in the ass.

Re: Dockerizing MySQL at Uber Engineering

#58

Post by user5994461 »

cwyers wrote: post #46
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

> 1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire. Really? The…

I think you're talking about different things.

MySQL only needs the my.cnf to start a server.

The permissions are stored along the database files (If i remember well), which is a separate problem. 1) It's only configured once on the master 2) Uber says that they don't reuse volumes [which implies many other consequences and special management around that].

By the way, Chef/Puppet/Ansible have build-in commands to manage that as well.

Re: Dockerizing MySQL at Uber Engineering

#59

Post by collyw »

falcolas wrote: post #22
vidarh wrote: post #16

Earlier quoted context omitted.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

Isn't that one advantage of Docker over VM's is that you have access to all of the RAM in a container while you don't in a VM? (I may well be wrong about that).

Re: Dockerizing MySQL at Uber Engineering

#60

Post by korzun »

VLM wrote: post #45

Most of the assumptions in the comments boil down to its a bad idea if you're resource constrained because you'll get much better performance when everythings on one big server with shared ram. However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it…

It's almost 2017. The scaling choices are not limited to Docker vs. bare metal.

With their budget, you should be able to clone and scale metal servers within minutes if you wanted to.

This is like Tumblr spending $50,000 grand a month on their developer AWS instances while not making any money. Cool but not very practical.

Post reply on HN