Live data from Hacker News

Dockerizing MySQL at Uber Engineering

eng.uber.com

11–20 of 107 posts

Re: Dockerizing MySQL at Uber Engineering

#11
post #6

It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary? I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...

"Single product" which consists of more than X thousands of micro-services.

>The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores

It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL

Re: Dockerizing MySQL at Uber Engineering

#12
post #5

It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?

It's gotta be the case. Probably choose trip db based on cutting map into squares or hexagons. Dump all location data from drivers and riders into it real time. Sounds like they've got batch jobs running against these databases to discover surge areas and stuff. Probably designed to only hit only 1-2 databases at a time for calculations they do a lot.

This really looks like a crap design overall for a company started well in the age of nosql and stream processing.

Re: Dockerizing MySQL at Uber Engineering

#14
post #6

Earlier quoted context omitted.

"Single product" which consists of more than X thousands of micro-services.

>The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL

Uber essentually built their own custom document store on top of MySQL. They explain their design and reasons (and why they didn't use Cassandra etc) in this post: https://eng.uber.com/schemaless-part-one/

Re: Dockerizing MySQL at Uber Engineering

#15
post #6

Earlier quoted context omitted.

"Single product" which consists of more than X thousands of micro-services.

>The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL

Which likely explains why they use their MySQL Schemaless engine[1].

1. https://eng.uber.com/schemaless-part-one/

Re: Dockerizing MySQL at Uber Engineering

#16
post #7

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

Re: Dockerizing MySQL at Uber Engineering

#18
post #5

It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?

It's gotta be the case. Probably choose trip db based on cutting map into squares or hexagons. Dump all location data from drivers and riders into it real time. Sounds like they've got batch jobs running against these databases to discover surge areas and stuff. Probably designed to only hit only 1-2 databases at a time for calculations they do a lot. This really looks like a crap design overall for a company started…

I'm going to guess your design assessment is unpopular. You may find that "old-skool" store-and-forward networks are pretty effective for slow-moving dimensions.

Re: Dockerizing MySQL at Uber Engineering

#19
post #14

Earlier quoted context omitted.

>The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL

Uber essentually built their own custom document store on top of MySQL. They explain their design and reasons (and why they didn't use Cassandra etc) in this post: https://eng.uber.com/schemaless-part-one/

Okay thanks, that explains the why but doesn't make it sound less terrible. They essentially built a nosql database on top of MySQL. Forgivable years ago but this was in 2014...

Re: Dockerizing MySQL at Uber Engineering

#20
This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and this approach will quickly start coming apart at the seams.

1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire.

2) Multiple MySQL processes on the same host wastes that host's resources. A single MySQL instance is perfectly capable of running multiple databases, and will respond faster because it will properly allocate the boxes memory according to each DB's usage. Multiple processes will each chomp up the configured bit of memory, not allowing individual databases to use the resources they need. It's always faster to serve data from memory than from disk (even if that disk is an SSD). Worse, under-utilized DB instances will be swapped off to disk, causing even more load and delay as they are swapped back in.

3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two. Neither will respond to requests with any alacrity, meaning you typically want to take both hosts out of the active DB pool.

4) The DB restoration process from copying over the raw files can take 10+ minutes, depending on how many dirty pages existed on the source. The restoration process will go faster with logical dumps, but logical dumps will take longer to generate, transfer, and load.

To reiterate, this are problems for those companies running at scale, with terabytes of data and dozens (or more) DB servers. When you're running a few GB of data, a DB container is probably going to work fine. Just don't believe for a second that you can scale it the same way you do your web frontend services.

It pays to hire experts. How much time and money has Uber sunk into working around their DB, instead of with it?

Post reply on HN