Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#61

Post by user5994461 »

memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

> This is a case where the db server should use the entire resources of a single server

They have thousands of clusters. They didn't design/architect anything.

They're likely just trying to regroup databases because they are heavily underutilized and noone knows WTF they are running. And the organization will keep growing like that, adding new databases every day.

Re: Dockerizing MySQL at Uber Engineering

#62

Post by evanelias »

otterley wrote: post #52
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

Additionally, the article appears to falsely assume that one can start a new container with a different and arbitrary MySQL version and reuse the data volumes from the previous container. From experience I know that this is not always the case: upgrades might work, but downgrades often will not. Even in the case of upgrades, various DB schema changes will be needed.

In my experience it's generally safe to upgrade (and usually even downgrade) across post-GA point releases of the same major version, i.e. 5.6.22 -> 5.6.31 or vice versa.

You should always run mysql_upgrade to be safe, and I'd assume Uber's automation does this. But you probably won't encounter problems if you don't.

mysql_upgrade typically just makes some quick changes to the system schema ("mysql" database), not to other data. And it typically only needs to change things across major version upgrades (5.5->5.6) or pre-GA releases; it's rare to make a change in a point release of a major version much after the GA release.

That all said, I don't ever advocate using Docker for MySQL in production at this time; see my longer comment in another subthread below for reasoning.

Re: Dockerizing MySQL at Uber Engineering

#63

Post by user5994461 »

Ruphin wrote: post #2

Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-co…

> Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state.

Docker is useless for what is not stateless.

You can run database in Docker... until the assumptions of Docker will come back to bite you.

> There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique.

Well. Run databases in Docker if you've got 3000 clusters and your only concern [that you created yourself] is to run multiple of them on the same hosts.

Re: Dockerizing MySQL at Uber Engineering

#64

Post by gaius »

drej wrote: post #3

I love these posts for several reasons. 1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real…

but it's interesting to see the engineering chops needed to maintain this operation

No. Uber has clearly hired too many engineers and has too little for them to do, leaving them free to come up with Rube Goldberg contraptions like this one. This is not anything any company that isn't swimming in money should even contemplate (and neither is slapping a layer on MySQL and claiming to have invented a "new datastore"). The kids are incharge of the nursery over at Uber and they are in desperate need of a good, adult CTO...

Re: Dockerizing MySQL at Uber Engineering

#65

Post by johnkoepi »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two...

It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

Re: Dockerizing MySQL at Uber Engineering

#66

Post by pryelluw »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

I miss being on a team with a proper DBA. Startups these days expect the full-stack dev to know everything under the sun. Your post was really great to read. Do you keep a blog?

Re: Dockerizing MySQL at Uber Engineering

#67

Post by toomuchtodo »

gaius wrote: post #64
drej wrote: post #3

I love these posts for several reasons. 1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real…

but it's interesting to see the engineering chops needed to maintain this operation No. Uber has clearly hired too many engineers and has too little for them to do, leaving them free to come up with Rube Goldberg contraptions like this one. This is not anything any company that isn't swimming in money should even contemplate (and neither is slapping a layer on MySQL and claiming to have invented a "new datastore"). T…

It doesn't surprise me that I had to scroll so far down in the thread to find this comment. Kudos for saying what needed to be said.

EDIT: Make your system as simple as possible, but no more simple than that.

Re: Dockerizing MySQL at Uber Engineering

#68

Post by evanelias »

korzun wrote: post #60
VLM wrote: post #45

Most of the assumptions in the comments boil down to its a bad idea if you're resource constrained because you'll get much better performance when everythings on one big server with shared ram. However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it…

It's almost 2017. The scaling choices are not limited to Docker vs. bare metal. With their budget, you should be able to clone and scale metal servers within minutes if you wanted to. This is like Tumblr spending $50,000 grand a month on their developer AWS instances while not making any money. Cool but not very practical.

Not sure if you were just using a made-up example, but Tumblr primarily isn't on EC2 and never has been. For compute they were bare-metal from the start and moved to their own self-managed datacenters in 2012.

Re: Dockerizing MySQL at Uber Engineering

#69

Post by didip »

I think the back story here is that they are running MySQL as sharded key-value storage.

And at 2300 instances, it makes sense that there would be a few of them dying at any given time.

With this setup, using MySQL is pretty much arbitrary, any databases could fill in the role as well. E.g. It got me thinking that Redis+Sentinel could do the job too (if they don't need super durable transactions)

Re: Dockerizing MySQL at Uber Engineering

#70

Post by user5994461 »

throwbsidbdk wrote: post #4

It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary? I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...

As said in a previous blog post by Uber, they have thousands of microservices and over 8000 Git repos.

It's safe to assume that their infra is a giant clusterfuck :D

I would assume that they scaled by adding more and more engineers, that end up working each in their corners on different problems, without any basic shared tooling/ practices/design. Things get out of hands quickly.

Post reply on HN