Dockerizing MySQL at Uber Engineering • Hacker News Board

Re: Dockerizing MySQL at Uber Engineering

#81

Post by falcolas » Mon, Nov 28, 2016, 5:15 PM UTC

pryelluw wrote: post #66

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

I miss being on a team with a proper DBA. Startups these days expect the full-stack dev to know everything under the sun. Your post was really great to read. Do you keep a blog?

I don't. If you really would like to learn more about the ins and outs of MySQL, though, I couldn't recommend the Percona blog higher.

The unfortunate thing, in my opinion, is that so much of being a software/system administrator of any flavor is having been burned by that software. That experience makes you invaluable, but it also makes you expensive to hire. Expensive hires are really hard for startups to justify.

That said, if you reach the size where you're having to shard your DB to hit performance metrics, you're at the point where it will typically cost you less to absorb that expensive hire (or consultant) than to attempt to naively engineer around the problem.

https://www.percona.com/blog/

Re: Dockerizing MySQL at Uber Engineering

#82

Post by privateprofile » Mon, Nov 28, 2016, 5:20 PM UTC

gaius wrote: post #64

drej wrote: post #3

I love these posts for several reasons. 1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real…

but it's interesting to see the engineering chops needed to maintain this operation No. Uber has clearly hired too many engineers and has too little for them to do, leaving them free to come up with Rube Goldberg contraptions like this one. This is not anything any company that isn't swimming in money should even contemplate (and neither is slapping a layer on MySQL and claiming to have invented a "new datastore"). T…

> The kids are incharge of the nursery over at Uber and they are in desperate need of a good, adult CTO...

That's a proper description of most startups I've had contact with recently, namely (well) funded startups.

It's a consequence of having inexperienced (and, sometimes, untalented) people running functions at a company.

It also applies to other areas like sales ("my inside sales script is killer, even though I haven't done professional sales anywhere before and just found out what inside sales is") or hiring ("I'll just hire my equally inexperienced buddies from college, they were great there").

But when it comes to kids building technology with "college playground" quality, it's too evident not to notice. The product ran fine when their buddies were testing, but it's unstable, unmaintainable and in need of a complete redesign when real world traffic comes along. All things that could have been prevented with a little bit of competence and experience.

It's a kind of mantra for people that take VC blogs as gospel and engineering blogs as the 10 commandments. Don't go to college, don't get any real world experience: start a company and, if it actually survives more than a couple of years, someone will inherit your technical debt.

This investor speak will quickly change when you get funded and have to attend board meetings to tell investors that your product is still not working, or you need twice the team that would be required for your goals, or you need months to produce basic business data...

Re: Dockerizing MySQL at Uber Engineering

#83

Post by GFischer » Mon, Nov 28, 2016, 5:23 PM UTC

buckhx wrote: post #76

falcolas wrote: post #20

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and…

I feel like a lot of their engineering blog posts are patting themselves on the back a bit prematurely. Here's an article I wrote last year about a blog post they put up about their approach to geofencing which I was not particularly impressed with https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...

Really interesting article. I'm also a bit naive when it comes to geofencing (but, then again, I'm not engineering in a 50bn company).

Waste of existing talent is unfortunately a way of life at large corporations :( .

Re: Dockerizing MySQL at Uber Engineering

#84

Post by user5994461 » Mon, Nov 28, 2016, 5:26 PM UTC

falcolas wrote: post #81

pryelluw wrote: post #66

Earlier quoted context omitted.

I miss being on a team with a proper DBA. Startups these days expect the full-stack dev to know everything under the sun. Your post was really great to read. Do you keep a blog?

I don't. If you really would like to learn more about the ins and outs of MySQL, though, I couldn't recommend the Percona blog higher. The unfortunate thing, in my opinion, is that so much of being a software/system administrator of any flavor is having been burned by that software. That experience makes you invaluable, but it also makes you expensive to hire. Expensive hires are really hard for startups to justify.…

> Expensive hires are really hard for startups to justify.

Or... startups are hard for experienced workers to justify.

Who would want to work for less money, shit equity, bad hours, on calls and surrounded by constant waves of variable level developers that you don't/can't get enough time to train.

Re: Dockerizing MySQL at Uber Engineering

#85

Post by falcolas » Mon, Nov 28, 2016, 5:29 PM UTC

vidarh wrote: post #72

falcolas wrote: post #22

Earlier quoted context omitted.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

Doesn't work that way with containers - they can all still have access to all the memory if you are confident you can safely give them access to it (and if you're not, then you certainly can't co-locate them in the same process. But I have yet to deal with a server where adding more RAM was more than a rounding error compared to getting a fast IO subsystem. And running them all on a single server means you need to ta…

> Doesn't work that way with containers - they can all still have access to all the memory

It's not a container problem I'm summing up here, it's a DB problem. Well behaved DB software won't grow to boundless limits to manage the workload, it will grow to it's configured limits.

Running multiple instances of a DB server means that each DB is going to be hard-coded to a limit which is some fraction of the available memory.

> I have yet to deal with a server where adding more RAM was more than a rounding error

There are limits even to this, though. The most memory you can get for any single, broadly available instance in AWS is 244 GiB, and its going to cost you an arm and a leg to run. It's not hard to create a dataset which exceeds 244 GiB in size.

> take down all of them to upgrade any one of them

Only if you're upgrading the actual DB software. And most time a master switch is sufficient to keep everything online through a DB version upgrade. Not to say that master switches are easy; but even Uber has that pain with their system.

> all of them being able to run on the same database version, and with the same extensions.

This isn't something I frequently see being an actual problem, within the same corporation.

Re: Dockerizing MySQL at Uber Engineering

#86

Post by user5994461 » Mon, Nov 28, 2016, 5:30 PM UTC

falcolas wrote: post #74

johnkoepi wrote: post #65

Earlier quoted context omitted.

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two... It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

Copying data is always going to be expensive, but it can't be avoided. The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB. The best you can do is be in control of when data transfers happens so you're doing…

I'd add that one can use throttling to bring new databases in live, while sustaining peak traffic.

Some databases have a configurable limit in MB/s for replication, or it's possible to assign disk/CPU quota on the slave to slow it down.

Combine that with good planning and monitoring, you'll be fine =)

Re: Dockerizing MySQL at Uber Engineering

#87

Post by falcolas » Mon, Nov 28, 2016, 5:33 PM UTC

collyw wrote: post #59

falcolas wrote: post #22

Earlier quoted context omitted.

Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip…

Isn't that one advantage of Docker over VM's is that you have access to all of the RAM in a container while you don't in a VM? (I may well be wrong about that).

It's not Docker that's the issue, it's that well behaved DB software limits its memory usage to a fixed (but configurable) value. Unless you hand-tune the size of the (in this case) InnoDB buffer pools for each DB instance, the memory usage will not be as efficient.

Of course, if you're hand-tuning different DB instances, you're already in trouble.

Re: Dockerizing MySQL at Uber Engineering

#88

Post by iamatworknow » Mon, Nov 28, 2016, 5:39 PM UTC

UK-AL wrote: post #33

memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

There are other benefits to docker than just being able to run multiple processes on the same machine. Putting everything you need to build an image using a dockerfile(Then to version control that) is useful. Developers can also grab docker images for databases for their localmachines without any setup. Also consistency with the rest of the stack.

If you don't need containerization and still want those features you mention why not just use Vagrant?

Re: Dockerizing MySQL at Uber Engineering

#89

Post by [deleted] » Mon, Nov 28, 2016, 6:48 PM UTC

user5994461 wrote: post #63

Ruphin wrote: post #2

Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-co…

> Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. Docker is useless for what is not stateless. You can run database in Docker... until the assumptions of Docker will come back to bite you. > There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. Well.…

[deleted]

Re: Dockerizing MySQL at Uber Engineering

#90

Post by UK-AL » Mon, Nov 28, 2016, 6:57 PM UTC

toomuchtodo wrote: post #51

UK-AL wrote: post #38

Earlier quoted context omitted.

And for a lot of people the tools of choice is docker...

Picking the wrong tool because its easier never really scales. Docker is not configuration management. Configuration management is not containerization. Disclaimer: DevOps by day. I care because it makes it easier for me (and cheaper for you) to come in and scale things if it was done right in the beginning (instead of me having to revamp your infrastructure to move components to the right tools), but either way I ge…

Most configuration management tools are used for managing docker on machines. I don't see them used much for setting up whats inside them.

Plus they don't match the convenience of just pulling down an image identical to production for developers(very important).

I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image.