Dockerizing MySQL at Uber Engineering
eng.uber.com
Dockerizing MySQL at Uber Engineering
1–10 of 107 posts
Re: Dockerizing MySQL at Uber Engineering
#2Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique.
We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-configuration of master-slave and federation setup, and automatic failover/recovery. It doesn't solve all the problems, and we still need to manually solve issues (mostly because we use MongoDB and ran into several bugs and 'features' over the years), but for standard operational work it saves a ton of work. Expanding our cluster by adding a new replicated shard is no more than 2 minutes of work in changing the master config, everything else is fully automated.
Re: Dockerizing MySQL at Uber Engineering
#3I love these posts for several reasons.
1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real impact on custormers. Everyday users of these technologies usually only get 'textbook' explanations, but blog posts like these allow them to actually find out how these can be used in production, what the caveats are, how they interplay with other parts of the stack etc. 3. Some people complain that when a startup gets bought or implodes, their technology virtually disappears instead of benefitting the wider developer community for learning purposes (I'm not here to argue what the best practice should be, just stating an observation of a common complaint). By continually describing their engineering practices (similarly to Airbnb in their engineering blog) and open sourcing their non-business technology (e.g. go-torch), their experience lives on beyond the life of the company.
Last but not least, it build a positive ethos about this company in the developer community.
Re: Dockerizing MySQL at Uber Engineering
#4 It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary?
I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...
Re: Dockerizing MySQL at Uber Engineering
#5It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?
Re: Dockerizing MySQL at Uber Engineering
#6It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary? I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...
"Single product" which consists of more than X thousands of micro-services.
Re: Dockerizing MySQL at Uber Engineering
#7> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations.
You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.
Re: Dockerizing MySQL at Uber Engineering
#8Screenshot from our management console.
This looks like an amazing tool.
Re: Dockerizing MySQL at Uber Engineering
#9> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.
Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?
Re: Dockerizing MySQL at Uber Engineering
#10> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.
Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?
And what gives you Docker what cgroups would not?
I mean they write:
Initially, all our clusters were managed by Puppet
They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run at AWS and could just use a AMI, but since that isn't written in the article I would not create such assumptions).