Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

21–30 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#21

Post by elcct »

kennethh wrote: post #5

It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?

Disclaimer: I have never used Uber.

What if user orders Uber from one city to another? If driver is then in another city, wouldn't be nice to give him some rides in that city or when someone wants to go back to original city? What about cities that are near the border? If user would like to go shopping to another country? (Not uncommon in Europe. Long time ago I used to take local bus from Poland to Germany, do some shopping and go back)

Re: Dockerizing MySQL at Uber Engineering

#22

Post by falcolas »

vidarh wrote: post #16
pmlnr wrote: post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Biggest reason not to run multiple database instances on a single bit of hardware - ram.

Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory.

On the flip side, if you have all five customers in one DB instance, then that one customer can house all 10gb of their data in working memory, making it much more performant.

You're hamstringing your DB processes by running one instance per set of databases. Run them all in one process, and you will get much more bang for your infrastructure buck.

Re: Dockerizing MySQL at Uber Engineering

#23

Post by falcolas »

saturn_vk wrote: post #9
pmlnr wrote: post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?

Memory. Why go to disk for reads in the first place, if your most used data is in memory? You can keep more of the hot data in memory if you use one process instead of two (or five, or ten).

Re: Dockerizing MySQL at Uber Engineering

#24

Post by memracom »

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool well, because you will be pushing the limits of server automation with a database server.

If you think that a database server is where you put your "application's database" then I question why you have a db server at all. That is the use case SQLITE is for. Database servers are for the use case of storing the business's mission critical data so that all the applications of the business can access needed data.

I love Docker and use it a lot, but not for everything.

Re: Dockerizing MySQL at Uber Engineering

#25

Post by icebraining »

elcct wrote: post #21
kennethh wrote: post #5

It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?

Disclaimer: I have never used Uber. What if user orders Uber from one city to another? If driver is then in another city, wouldn't be nice to give him some rides in that city or when someone wants to go back to original city? What about cities that are near the border? If user would like to go shopping to another country? (Not uncommon in Europe. Long time ago I used to take local bus from Poland to Germany, do some…

Sounds fairly simple (which is not to say easy to implement); you keep the car tied to the database of origin for the duration of the ride (other users don't need to know about it while it's occupied), then you can migrate the car to the other DB when it's idle.

Re: Dockerizing MySQL at Uber Engineering

#26

Post by aeharding »

memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

Eh. For small scale, running a DB in a container and linking the DB storage to a directory on the host's filesystem works well enough.

I can't talk about large scale since I don't have first hand experience managing a setup like that.

Re: Dockerizing MySQL at Uber Engineering

#27

Post by memracom »

Ruphin wrote: post #2

Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-co…

If you would use something like Ansible it would be just as fast because it is fully automated, and there would be no work at all to change a config because Ansible will also customize config files from templates. Chef and Puppet and SaltStack work similarly, just learn how to use your tool of choice.

Re: Dockerizing MySQL at Uber Engineering

#28

Post by liveoneggs »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

5) configuration and version drift is guaranteed since you have to restart databases to make any changes

5a) ignores MySQL's ability to make many runtime config changes without a restart

6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json)

7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment)

8) requires additional ops overhead (noted as special care needed for masters)

---

But if you already run everything in docker and really want to get rid of puppet, then why not?

---

this article links to ansible but not puppet and is the only blog post tagged with either one; I think this really boils down to puppet vs $x

Re: Dockerizing MySQL at Uber Engineering

#29

Post by ec109685 »

falcolas wrote: post #22
vidarh wrote: post #16

Earlier quoted context omitted.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

The limitation of the approach you are advocating is that all databases would need to replicate from the same master, which isn't always practical.

Re: Dockerizing MySQL at Uber Engineering

#30

Post by [deleted] »

saturn_vk wrote: post #9
pmlnr wrote: post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?

[deleted]
Post reply on HN