Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

eng.uber.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#11

Post by fweespeech »

pbreit wrote: post #4

Anyone want to weigh in on whether or not Postgres is a viable option for this?

> Since Schemaless uses MySQL asynchronous replication, the write will be lost if a master receives a write request, persists the request, and then fails before it has replicated the write to the minions (e.g., in a hard drive failure). To solve this problem we use a technique called buffered writes. Buffered writes minimize the chance of losing data by writing it to multiple clusters. If a master is down, the data is not readily available for subsequent reads but has nevertheless been persisted.

Postgres has the same replication semantics as MySQL async replication. So yes, if you want to build something like this, you could.

However, the question you should be asking yourself isn't "could you" but "should you".

The answer is probably not.

Multi-master setups are extremely difficult to build reliably and unless you understand them extremely well you can frequently shoot yourself in the foot with something that seems right but really is only right for a handful of use cases.

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#12

Post by cwyers »

jrcii wrote: post #6
pbreit wrote: post #4

Anyone want to weigh in on whether or not Postgres is a viable option for this?

Only semi-related, I've discerned a Postgres > MySQL/MariaDB sentiment on HN for the last year or so. Is that just my imagination? If not, why is that? MySQL in my experience is a very powerful RDBMS, maybe I just haven't run up against its limitations.

The short version is, Postgres is developed under the "do it right, then make it fast" philosophy, for the most part. Early versions of MySQL were like, "ACID? Pbbbbbbbbbt." So they treated your data like it was essentially worthless and it could lose it at about any time. For a lot of applications built on hardware at the time and the needs at the time, the data really was kinda worthless and so people built stuff on it anyway. But a lot of people remember those early days when MySQL was a pretty crappy database that just happened to be fast and scale out well. MySQL has mostly caught up to Postgres in the whole "not losing data" sense, and is catching up in terms of SQL features as well (although I wish it had CTEs, among many other things). Postgres has made a lot of advances in speed and scalability too, but still lacks some of what MySQL has in terms of clustering and the like without using plugins or add-ons.

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#15

Post by morgo »

austinhyde wrote: post #10
jrcii wrote: post #6

Earlier quoted context omitted.

Only semi-related, I've discerned a Postgres > MySQL/MariaDB sentiment on HN for the last year or so. Is that just my imagination? If not, why is that? MySQL in my experience is a very powerful RDBMS, maybe I just haven't run up against its limitations.

There's probably a couple reasons, but I'll summarize my experiences with them. I haven't dug into MySQL for a few years (last I really worked with it was 5.5), but out of the box, it does a lot of things that are pretty unsafe or encourage bad practices, such as truncating data when it's longer than the column size, inserting the zero-value for a NOT NULL column rather than erroring when a NULL is inserted, confusin…

> I haven't dug into MySQL for a few years (last I really worked with it was 5.5), but out of the box, it does a lot of things that are pretty unsafe or encourage bad practices, such as truncating data when it's longer than the column size, inserting the zero-value for a NOT NULL column rather than erroring when a NULL is inserted, [..]

This is no longer the default in MySQL 5.7.

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#16

Post by jkovacs »

danbruc wrote: post #13

Color me skeptical, that looks like a pretty strange design to me, a database on top of a database [1]. [1] http://c2.com/cgi/wiki?GodTable

Agreed, the design immediately reminded me of the following ancient DailyWTF article, one of the few ones that stuck in my head ever since I read it. They called it the "Inner-Platform Effect" - might apply here.

http://thedailywtf.com/articles/The_Inner-Platform_Effect

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#17

Post by dimfeld »

yid wrote: post #5
pbreit wrote: post #4

Anyone want to weigh in on whether or not Postgres is a viable option for this?

Might be a case of MySQL natively supporting multi-master replication, whereas with Postgres you have to use a third-party/commercial solution. Especially if you're not using any particularly advanced Postgres features, the operational simplicity of having built-in multimaster replication might outweigh any PG benefits.

Are they actually using multi-master replication? I gleaned from the article series that they have many small independent MySQL clusters, each with one master and two slaves, and that their worker layer handles requesting data from the correct cluster for whatever shard a request routes to. Writes to a cluster can only go the master. So this seems more like a single-master design, just replicated a bunch but without the masters communicating with each other.

That said, I haven't used MySQL much at all. Am I just missing something or misunderstanding what multi-master means in the context of MySQL?

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#18

Post by collyw »

frik wrote: post #9
pbreit wrote: post #4

Anyone want to weigh in on whether or not Postgres is a viable option for this?

Can you please stop? Postgres is great and all. But people use MySQL for valid reasons, web scale is one. And InnoDB is a very very good DB engine for that task.

"web scale is one"

Does "web scale" actually mean anything? I thought it was just a piss take from the "MongoDb is web scale" cartoon.

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#19

Post by abiox »

collyw wrote: post #18
frik wrote: post #9

Earlier quoted context omitted.

Can you please stop? Postgres is great and all. But people use MySQL for valid reasons, web scale is one. And InnoDB is a very very good DB engine for that task.

"web scale is one" Does "web scale" actually mean anything? I thought it was just a piss take from the "MongoDb is web scale" cartoon.

i don't think people actually use the expression "web scale" seriously, but i can't tell if they're joking. poe's law or something.

Re: The Architecture of Schemaless, Uber Engineering’s Trip Datastore Using MySQL

#20

Post by ivansavz »

I've been following these Uber engineering articles, and I think this is a very neat architecture. Append only + boring technology = solid stuff.

I'm curious to know how many shards per storage cluster they use and how this mapping is done. Is it fixed or can it change? I imagine a startup trying to use a similar setup could start with a few storage clusters, but then add more clusters as needs grow...

They say they use 4096 shards (presumably generated based on some part of `row_key` which is the trip id), but I'm not sure this is a generally-applicable strategy. e.g. if sharding in a social netowrk website is performed based on `user_id` then won't be able to do joins across `user_id`s.

Post reply on HN