Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

31–40 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#31

Post by ec109685 »

merb wrote: post #10
saturn_vk wrote: post #9

Earlier quoted context omitted.

Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?

And what gives you Docker what cgroups would not? I mean they write: Initially, all our clusters were managed by Puppet Of course docker won't actually replace such systems. Uber has the worst enginnering practices i've ever seen. They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run…

> I would not create such assumptions

But you assume they have the worst engineering practices based on a few blog posts?

Re: Dockerizing MySQL at Uber Engineering

#32

Post by falcolas »

liveoneggs wrote: post #28
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additi…

> But if you already run everything in docker and really want to get rid of puppet, then why not?

And you know, I wouldn't argue too much at the thought of running MySQL out of a container. It does avoid a lot of basic issues around upgrades, versioning, stack split, etc. It's the act of treating it like all other containerized software which gets my panties in a twist.

Stateful systems can't be treated like stateless systems if you want to maintain anything resembling reasonable uptime and performance.

Re: Dockerizing MySQL at Uber Engineering

#33

Post by UK-AL »

memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

There are other benefits to docker than just being able to run multiple processes on the same machine.

Putting everything you need to build an image using a dockerfile(Then to version control that) is useful. Developers can also grab docker images for databases for their localmachines without any setup. Also consistency with the rest of the stack.

Re: Dockerizing MySQL at Uber Engineering

#34

Post by falcolas »

ec109685 wrote: post #29
falcolas wrote: post #22

Earlier quoted context omitted.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

The limitation of the approach you are advocating is that all databases would need to replicate from the same master, which isn't always practical.

Given the choice between performance and ease of replication, the businesses I worked for always chose performance at scale. Perhaps that skews my expectations, but given how much DB performance affects overall app performance, it trumps most other considerations.

Re: Dockerizing MySQL at Uber Engineering

#36

Post by merb »

liveoneggs wrote: post #28
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additi…

> But if you already run everything in docker and really want to get rid of puppet, then why not?

Docker does not replace puppet and they clearly wrote that they developed a System just for that.

Re: Dockerizing MySQL at Uber Engineering

#37

Post by alex_hitchins »

vidarh wrote: post #16
pmlnr wrote: post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Multi DB hosting isn't the issue, it's installing two MySql server instances that I think people are questioning.

I know with MSSQL there were scenarios where you would install multiple instances, I think they were when you had more RAM in the box than the server package was licenced for, or for management of a large number of databases.

Re: Dockerizing MySQL at Uber Engineering

#38

Post by UK-AL »

memracom wrote: post #27
Ruphin wrote: post #2

Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-co…

If you would use something like Ansible it would be just as fast because it is fully automated, and there would be no work at all to change a config because Ansible will also customize config files from templates. Chef and Puppet and SaltStack work similarly, just learn how to use your tool of choice.

And for a lot of people the tools of choice is docker...

Re: Dockerizing MySQL at Uber Engineering

#39

Post by agentgt »

liveoneggs wrote: post #28
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additi…

I find it ironic that one of Uber's original impetus for migrating off Postgres (which I still have serious doubts about) was not being able to "trust" the system [1].

Given my limited failed experience with playing with docker + postgres (and various volume stores including flocker) I have continued doubts about Uber's choices.

I suppose they are having success but I wonder at what cost.

Specifically the quote on [1]:

Finally, our decision ultimately came down to operational trust in the system we’d use, as it contains mission-critical trip data. Alternative solutions may be able to run reliably in theory, but whether we would have the operational knowledge to immediately execute their fullest capabilities factored in greatly to our decision to develop our own solution to our Uber use case. This is not only dependent on the technology we use, but also the experience with it that we had on the team.

[1]: https://eng.uber.com/schemaless-part-one/

Re: Dockerizing MySQL at Uber Engineering

#40

Post by samfisher83 »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

I think there is cyclicity to software development. It starts at some point it goes to greater abstraction and then when it gets a to point it becomes more about getting closer to metal to improve performance and then that cycle skews toward abstraction again.
Post reply on HN