Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#101

Post by pg314 »

user5994461 wrote: post #99
pg314 wrote: post #98

Earlier quoted context omitted.

> As a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea. Storage capacity is also growing exponentially: Samsung is shipping a 15TB SSD (albeit at $10000), Seagate has previewed a 60TB SSD. > You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required. The limits of RDBMs are well understood. You don't need an…

If you can't get the hardware neither on AWS nor Google nor SoftLayer. I'd consider that exotic enough. Don't get me wrong. I know vertical scaling and I've done it before. I'd take an old school DBA who understands Oracle over a random junior speaking only NoSQL to everything. For the majority of use cases (including where I am now), it's easier to pick the right technology (Cassandra) even if we have to learn and l…

> Of course, if you have extensive experience with PostgreSQL, that may skew the choice heavily to the other direction ;)

I have more experience with PostgreSQL than Cassandra, for sure. And I've had negative experiences with people trying to push Cassandra where it was totally not appropriate (small problem and no need for high availability). They had no experience with Cassandra themselves beyond watching a video and doing some tutorials and couldn't answer basic questions about the underlying technology. That might be skewing my perception too.

Re: Dockerizing MySQL at Uber Engineering

#102

Post by toomuchtodo »

UK-AL wrote: post #90
toomuchtodo wrote: post #51

Earlier quoted context omitted.

Picking the wrong tool because its easier never really scales. Docker is not configuration management. Configuration management is not containerization. Disclaimer: DevOps by day. I care because it makes it easier for me (and cheaper for you) to come in and scale things if it was done right in the beginning (instead of me having to revamp your infrastructure to move components to the right tools), but either way I ge…

Most configuration management tools are used for managing docker on machines. I don't see them used much for setting up whats inside them. Plus they don't match the convenience of just pulling down an image identical to production for developers(very important). I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image.

> I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image.

Which is why you should be provisioning with Vagrant locally to properly replicate production.

Re: Dockerizing MySQL at Uber Engineering

#103

Post by vidarh »

falcolas wrote: post #100
vidarh wrote: post #92

Earlier quoted context omitted.

> Running multiple instances of a DB server means that each DB is going to be hard-coded to a limit which is some fraction of the available memory. What you are suggesting then, is setting limits in a way that does not guarantee resources for any given database. > There are limits even to this, though. The most memory you can get for any single, broadly available instance in AWS is 244 GiB, and its going to cost you…

> does not guarantee resources for any given database Correct. Instead it gives the most resources to the pages in all of the databases which are accessed and used the most. MySQL and InnoDB are remarkably well tuned to ensure that the most often used pages are in memory, where they can be accessed and updated with the greatest performance. > It's something I see all the time. We apparently work on very different use…

> Instead it gives the most resources to the pages in all of the databases which are accessed and used the most.

And that's not acceptable when dealing with setups where each database needs predictable performance, and some of them may see heavy traffic while others don't.

> Just be aware that your usecase is far from typical.

I don't think there is such a thing as a "typical" usecase in this area based on what I see as part of my consulting work, but this is common enough that I deal with it regularly across a wide variety of clients.

Re: Dockerizing MySQL at Uber Engineering

#104

Post by memracom »

otterley wrote: post #54
memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

I wouldn't go so far as to say they should "never be containerized," but doing so doesn't solve any of your problems other than a packaging problem, and it arguably creates problems you might not have had before. It's my experience that most of the impetus behind Docker is driven by developers who need it on their Mac/Windows workstations to run Linux processes, and who (for good reason) want to ensure that their dev…

I also develop on OSX or Windows systems, targeting Linux as the deployment system. And I also want to test the code in an environment as close to production as possible. That is why I install Virtualbox and set up virtual Linux servers using the exact same server management scripts (in my case Ansible) as are used to configure production servers. It works great with no Docker. But I do use Docker as well for servers that are more cookie cutter than a db. For instance a JVM app server that has two Docker images, one with 3 layers culminating in the JVM app, and one with 2 layers culminating in an NGINX SSL proxy.

And I don't run Docker on OSX at all because it is just a wrapper for running Virtualbox with a Linux server inside. It is simplicity to just set up Virtualbox directly and just use Linux virtual servers directly.

Re: Dockerizing MySQL at Uber Engineering

#105

Post by memracom »

UK-AL wrote: post #96
memracom wrote: post #24

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool wel…

Why? There are tons of advantages of using docker that's not just for running multiple processes on a single machine. You can still gain advantages from just installing a single database container on a big server + docker volumes. Being able to version control image creation scripts, being able to pull a identical image for both dev/production is useful and quicker than trying to use puppet on dev machines, the fact…

I don't understand your puppet comment, but then I have never used it. I had some experience with Chef, lots of experience with shell scripts to config servers from scratch. But now I use Ansible instead. Technically I still run a shell script to call AWS CLI and launch an instance, but then I do everything else with an Ansible playbook that configures the server, installs a specific version of PostgreSQL, adds some extras like server extensions and a performance reporting tool. This is what set up our current production and dev servers, and from time to time I use it to set up yet another server to test something like adding a language extension, before running it against the dev server.

And as I mentioned, I also use Docker for other servers. My experience is that using Docker to set up a server is usually the same amount of work as Ansible. But Ansible is more amenable to refactoring into reusable components, i.e. components of the build/config process.

Re: Dockerizing MySQL at Uber Engineering

#106

Post by seschwar »

falcolas wrote: post #74
johnkoepi wrote: post #65

Earlier quoted context omitted.

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two... It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

Copying data is always going to be expensive, but it can't be avoided. The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB. The best you can do is be in control of when data transfers happens so you're doing…

That sounds a bit like what Joyent is doing in their Autopilot Pattern implementation for MySQL: https://www.joyent.com/blog/dbaas-simplicity-no-lock-in

Re: Dockerizing MySQL at Uber Engineering

#107

Post by derekperkins »

evanelias wrote: post #47
pmlnr wrote: post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

I strongly disagree. At large scale, there are huge advantages to running multiple mysqld on one physical host. Facebook does this across their entire DB fleet! Most of their DB hosts run 2 mysqld but some run 8 or more -- it depends on which workload the host is part of. The original motivation was to support multiple hardware generations. PCIe flash cards have become much larger over the years very quickly -- much…

YouTube already open sourced their containerized MySQL solution and it works pretty great. http://vitess.io
Post reply on HN