Live data from Hacker News

Why Uber Engineering Switched from Postgres to MySQL

eng.uber.com

51–60 of 306 posts

Re: Why Uber Engineering Switched from Postgres to MySQL

#51
I would argue that most of these Postgres "flaws" are actually advantages over MySQL when you look at them holistically rather than the very specific Uber use-case.

Postgres's MVCC is superior (can rollback DDL, can add indexes online, can have open read transactions for a VERY long time without impacting other parts of the system)

Postgres supports many types of indexes, not just b-tree. One thing it doesn't have is clustered b-tree indexes... which is really what MySQL does that makes it somewhat "better." I wonder how Uber adds an index to a table that already has 1B+ rows in it with mysql?

Postgres have WAL level replication is a better guarantee of actually replicating the data correctly. I cannot tell you how many times I've had to tell my boss that the "mysql replicas might be slightly out of sync with the master" because of various replication issues. The way it handles triggers and scheduled events alone is garbage and can very easily break replication and/or silently cause inconsistency.

As for data corruption, if there is a bug that causes corruption, then there is a bug. I don't think that is a fundamental design flaw as implied in this article. You shouldn't rely on 1/2 assed replication design to accidentally save you from the data corruption bug. There are many downsides to the design MySQL has that are simply not listed here.

I have been both a professional MySQL administrator as well as Postgresql (as well as SQL Server and many NoSQL engines). Many of these Postgres issues are only issues at crazy huge scale, and I would say at that point you probably want to move away from relational anyway. MySQL has its own very large set of problems at scale as well.

It sounds like Uber is using MySQL as just a data bucket with primary keys ("Schemaless") which is good -- because you can't alter tables to save your life with MySQL.

At the end of the data each developer/business needs to use what works for them, but I would really shy away from pointing to this article as a linchpin in the "MySQL vs. Postgres" war (if there even is such a thing.)

Re: Why Uber Engineering Switched from Postgres to MySQL

#52
Interesting post! While I suspect that a MySQL installation is just as likely to have its own problems in the long run, I'm not smart enough to provide any kind of compelling point-by-point refutation. However, a number of the points made strike me as having possible trade-offs that were not really addressed in-depth.

My summary of the arguments against Postgres and some basic thoughts on each:

1. Writes are more expensive because all secondary indexes must be updated with a new physical location.

This may be true, but the MySQL model of using primary keys from secondary indexes will mean that reads are inherently expensive. They even mention this:

> This design means that InnoDB is at a slight disadvantage to Postgres when doing a secondary key lookup, since two indexes must be searched with InnoDB compared to just one for Postgres.

So it seems like a classic read vs. write trade-off.

I'm also a little skeptical of any performance claims that don't include any numbers. It's possible that efficient coding in Postgres makes this much more of a wash in terms of performance than claimed here.

2. Replication is less efficient because it's sending a lot of physical information out along the stream.

This is quite true, but IMO unlikely to be a major issues for most users unless they're dealing with a huge amount of data and streaming it over a slow connection (i.e. across the continent like Uber's disaster recovery center).

3. Data corruption from a bug found in 9.2.

Certainly a bad situation, but IMO not really a valid claim for situation. 9.2 is way behind at this point, and there's not much to say that they wouldn't have encountered a similar bug or something worse in MySQL in all that time, especially operating at scale.

To give a counter-anecdote, I operated Postgres at scale for a long time across many versions starting at 9.1 and was lucky enough to have never once encountered a bug with data corruption.

4. Postgres' MVCC model makes it easy for replicas to accidentally fall behind their master.

This one is valid (and annoying), but there are very good reasons for it, and you have some switches to control the behavior based on value transactions finishing on followers or prompt replication more highly.

5. Upgrades are difficult because the WAL stream works at a physical level and is not compatible between database versions.

Again, this is valid, but the statement-based replication is a scary idea. Row-level replication is more interesting and probably something that Postgres should have though.

Some good news is that Postgres is getting closer to logical WAL streaming, which should make in-place upgrades possible.

Re: Why Uber Engineering Switched from Postgres to MySQL

#53

I wonder what the design decisions are behind (or what it would take) to make Postgres store secondary indexes on disk like InnoDB does. Sure, the extra index lookup through the primary index is a cost, but it seems like write-amplification can sure be a greater concern too. Ultimately, it would be nice if Postgres gave the DBA a choice of — if not move outright to — secondary-index indirection through the primary in…

I would definitely not want PostgreSQL to "move to" the mechanism used by InnoDB as that mechanism is slower for reads (which this article even admits). This disk layout is one of the reasons I continue to use PostgreSQL. If you care more about write performance than reads you might even want to look into something entirely different than a traditional database.

FWIW, PostgreSQL's mitigation to the write amplification problem mentioned here are "heap only tuples". It is highly likely that Uber could have prevented a lot of this pain by learning more about this feature--which is notably not mentioned even once in this entire article, which to me completely undermines the feel they are trying to achieve of "we really really really know what we are doing"--and tuning their table density parameters to take maximal advantage.

Instead of "why we moved off of X" it would be much better to see "we are considering moving off of X: anyone know what we are doing wrong?". Sure, maybe Uber knows about HOT, and did extensive analysis to determine it wasn't a solution to their problem; but it frankly does not seem at all to be the case. Anyone who knows a lot about PostgreSQL would have explained HOT to them, so they probably didn't even consult with PostgreSQL experts behind the scenes.

Sadly, "we are looking for help with a complex technical challenge" is something the market punishes under the premise that everyone has to be entirely self-sufficient gods of their technology stack :(. The only time I remember ever having seen a company reach out to the community for help was reddit (with respect to something involving PostgreSQL or Cassandra... I don't remembee the specific issue).

Re: Why Uber Engineering Switched from Postgres to MySQL

#54
post #21

Earlier quoted context omitted.

pgbouncer works pretty well for that. I kinda wish it was part of the default Postgres install.

How does pgbouncer work with Amazon RDS (where you can't install separate software)?

you install it on the clients of the servers themselves and "connect locally" to the bouncer or have a bouncer step (server pair) in front of the database connections themselves.

Re: Why Uber Engineering Switched from Postgres to MySQL

#55

Earlier quoted context omitted.

I strongly disagree. it would have been useful f they'd stuck to problems without well-known solutions. Sadly, they also mixed in issues which are easily solved, or in a particularly egregious case, where they just complain about a bug. As though MySQL never had a bug. That was silly. My read of it was: Postgres annoyed us a few times, and we got fed up with its, so now something different will annoy us. Please look…

Almost all problems are already solved. "Solving" a problem today is mostly ego-stroking.

Sweet, I have a few PhD thesis and NSF grant proposals I'd like your help on.

Re: Why Uber Engineering Switched from Postgres to MySQL

#56

Earlier quoted context omitted.

From the post: > [...] This design difference means that the MySQL replication binary log is significantly more compact than the PostgreSQL WAL stream. Doesn't sound like what you described at all.

so you're pulling one line from the article to tell me that i'm wrong? on-disk format/write amplification: > For tables with a large number of secondary indexes, these superfluous steps can cause enormous inefficiencies. For instance, if we have a table with a dozen indexes defined on it, an update to a field that is only covered by a single index must be propagated into all 12 indexes to reflect the ctid for the new…

>so you're pulling one line from the article to tell me that i'm wrong?

Better than telling them that they are wrong while not only not pulling even one line from their article, but misattributing it to be something very different from what it is.

In what world does the response to the concerns and analysis in the article can ever be: "data corruption: everyone has bugs"...

Re: Why Uber Engineering Switched from Postgres to MySQL

#57
post #8

Earlier quoted context omitted.

now sure MySQL qualifies as "new" or "shiny" or "toy"

but getting to use "Schemaless" and "microservices" so that they could "shard"

Because sharding and schemaless are some recent novelties, huh?

(And even microservices are established practices -- heck Amazon was built on them more than a decade ago).

Re: Why Uber Engineering Switched from Postgres to MySQL

#58
Roadmaps ( PostgreSQL ) 2016-2017-...

* Postgres Professional roadmap ( Pluggable storages, Multimaster cluster with sharding, Effective partitioning, Adaptive query planning, Page-level data compression, Connection pooling, Native querying for jsonb with indexing support, ....) https://wiki.postgresql.org/wiki/Postgres_Professional_roadm...

* EnterpriseDB database server roadmap ( Parallelism, Replication, Vertical Scalability, Performance ) https://wiki.postgresql.org/wiki/EnterpriseDB_database_serve...

====

And "Scalable PostgreSQL for real-time workloads https://www.citusdata.com " --> https://github.com/citusdata/citus

Re: Why Uber Engineering Switched from Postgres to MySQL

#59

This was a great overview and write-up. Anyone know why they are using MySQL over MariaDB[1]? 1. https://mariadb.org/

I'm not saying this is the case at Uber, but it's quite common for MariaDB users to refer to it as MySQL, or to use the two names interchangeably. It's difficult to get out of the habit as all of the CLI tools are still mysql*, clients (i.e. the protocol) are all named "mysql", CM modules (like Ansible's) are all named "mysql", etc. And sometimes it's just simpler (if lazier) to say you're using MySQL rather than explain what MariaDB is and why it's preferable.

Re: Why Uber Engineering Switched from Postgres to MySQL

#60

The connection handling section was surprising to me, reading that Postgres uses a process per connection! This is pretty shocking to me, in a bad way.

Different design goals, Mysql lot of connections with shared memory model and fast running small queries, ex: PHP one connection per page view. Postgres dedicated memory model long running complex queries.
Post reply on HN