Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#71

Post by vidarh »

alex_hitchins wrote: post #37
vidarh wrote: post #16

Earlier quoted context omitted.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Multi DB hosting isn't the issue, it's installing two MySql server instances that I think people are questioning. I know with MSSQL there were scenarios where you would install multiple instances, I think they were when you had more RAM in the box than the server package was licenced for, or for management of a large number of databases.

Great, except when different customers requires different versions or different extensions.

Re: Dockerizing MySQL at Uber Engineering

#72

Post by vidarh »

falcolas wrote: post #22
vidarh wrote: post #16

Earlier quoted context omitted.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

Doesn't work that way with containers - they can all still have access to all the memory if you are confident you can safely give them access to it (and if you're not, then you certainly can't co-locate them in the same process.

But I have yet to deal with a server where adding more RAM was more than a rounding error compared to getting a fast IO subsystem.

And running them all on a single server means you need to take down all of them to upgrade any one of them, and are contingent on all of them being able to run on the same database version, and with the same extensions.

Re: Dockerizing MySQL at Uber Engineering

#73

Post by vidarh »

collyw wrote: post #59
falcolas wrote: post #22

Earlier quoted context omitted.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

Isn't that one advantage of Docker over VM's is that you have access to all of the RAM in a container while you don't in a VM? (I may well be wrong about that).

Yes, you can. If anything, one of the downsides of Docker is that vs. e.g. OpenVz or full VMs the memory accounting is still much less mature and requires additional work to enable on many distros.

Re: Dockerizing MySQL at Uber Engineering

#74

Post by falcolas »

johnkoepi wrote: post #65
falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two... It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

Copying data is always going to be expensive, but it can't be avoided.

The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB.

The best you can do is be in control of when data transfers happens so you're doing it when it makes sense and not in the middle of your highest traffic period (which is what frequently happens when attempting to automatically scale DBs in response to load).

Re: Dockerizing MySQL at Uber Engineering

#75

Post by falcolas »

evanelias wrote: post #62
otterley wrote: post #52

Earlier quoted context omitted.

Additionally, the article appears to falsely assume that one can start a new container with a different and arbitrary MySQL version and reuse the data volumes from the previous container. From experience I know that this is not always the case: upgrades might work, but downgrades often will not. Even in the case of upgrades, various DB schema changes will be needed.

In my experience it's generally safe to upgrade (and usually even downgrade) across post-GA point releases of the same major version, i.e. 5.6.22 -> 5.6.31 or vice versa. You should always run mysql_upgrade to be safe, and I'd assume Uber's automation does this. But you probably won't encounter problems if you don't. mysql_upgrade typically just makes some quick changes to the system schema ("mysql" database), not to…

> In my experience it's generally safe to upgrade (and usually even downgrade) across post-GA point releases of the same major version

My experience with this mirrors the parents. The more data you have, and the more obscure engine features you use, the more likely it is to happen. It only takes one major failure to make you want to test the daylights out of any upgrades or downgrades.

Re: Dockerizing MySQL at Uber Engineering

#76

Post by buckhx »

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

I feel like a lot of their engineering blog posts are patting themselves on the back a bit prematurely. Here's an article I wrote last year about a blog post they put up about their approach to geofencing which I was not particularly impressed with https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...

Re: Dockerizing MySQL at Uber Engineering

#77

Post by [deleted] »

falcolas wrote: post #74
johnkoepi wrote: post #65

Earlier quoted context omitted.

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two... It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

Copying data is always going to be expensive, but it can't be avoided. The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB. The best you can do is be in control of when data transfers happens so you're doing…

[deleted]

Re: Dockerizing MySQL at Uber Engineering

#79

Post by evanelias »

falcolas wrote: post #75
evanelias wrote: post #62

Earlier quoted context omitted.

In my experience it's generally safe to upgrade (and usually even downgrade) across post-GA point releases of the same major version, i.e. 5.6.22 -> 5.6.31 or vice versa. You should always run mysql_upgrade to be safe, and I'd assume Uber's automation does this. But you probably won't encounter problems if you don't. mysql_upgrade typically just makes some quick changes to the system schema ("mysql" database), not to…

> In my experience it's generally safe to upgrade (and usually even downgrade) across post-GA point releases of the same major version My experience with this mirrors the parents. The more data you have, and the more obscure engine features you use, the more likely it is to happen. It only takes one major failure to make you want to test the daylights out of any upgrades or downgrades.

I agree; I wasn't advocating not testing. My point was more that the parent was overstating the risk, particularly around things like required schema changes, or data volumes no longer being usable. Those particular things aren't issues in my experience (for post-GA point releases), and I've worked at crazy scale.

The more common classes of problem are performance degradation, config options being added or renamed, new features being buggy in rare edge cases, etc. Not things particular to InnoDB's storage format or things that would prevent a volume from being usable with a different version.

Re: Dockerizing MySQL at Uber Engineering

#80

Post by user5994461 »

I guess we should train the new generation better so they don't end up as Uber so... tip for any company who gets incredibly successfull:

The first thing one has to do is to drop SQL databases as the main data source.

The usual choice is to move to Cassandra. It does have build-in sharding AND backup AND multi-master replication AND multi datacenter support AND performances scale linearly with the number of servers.

The [only] other option is ElasticSearch (which has slightly different properties regarding data format and data consistence).

You don't need to craft complex custom sharded distributed WTF software to abstract hundreds of cluster of hundreds of databases. Use the right tool for the job, that has that build in.

Post reply on HN