Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Uber Engineering Switched from Postgres to MySQL (2016)

eng.uber.com

Post reply on HN
Print view View original

81–90 of 133 posts

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#81

Post by evanelias »

throwdbaaway wrote: post #21
glogla wrote: post #16

Earlier quoted context omitted.

Sure, but it does not really matter. Uber has not switched from Postgres used as RDBMS to MySQL used as RDBMS , they switched from Postgres used as RDBMS to MySQL used as key-value storage layer of homegrown sharded non-relational database . This has pretty much no bearing on anyone using Postgres or MySQL in reasonable way.

Exactly. I think the prior HN discussions failed to call out how painful it is to do any sort of schema migration against a big InnoDB table [1][2]. Well known MySQL uses such as Facebook TAO and this Uber Schemaless are typically abstractions built on top of MySQL, which means the schemas are pretty much static, and they don't feel the schema migration pain. For a typical RoR startup that relies on a RDBMS, please,…

To echo and add to rwultsch's sibling comment:

* Facebook had extremely frequent schema changes, and powerful declarative schema management automation to support this

* The TAO (or more correctly "UDB") use-case supported using many separate tables, not one giant generic key/value table as people seem to assume

* The non-UDB MySQL use-cases at Facebook, in combination, are still larger than the vast, vast majority of all other companies' databases. These non-UDB databases use a wide range of MySQL's functionality. The frequent claims that "Facebook used MySQL just as a dumb K/V store" are absolutely incorrect and have never been correct.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#82

Post by evanelias »

corty wrote: post #14
isoprophlex wrote: post #11

Earlier quoted context omitted.

That doesn't exactly sell me on MySQL, sounds like a recipe for disaster

There is more, however not everything is still valid or valid for every table type in MySQL: https://sql-info.de/mysql/gotchas.html

"This page deals with issues related to MySQL 4.1 and earlier, not 5.0"

MySQL 5.0 came out over 15 years ago.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#83

Post by junon »

There's some historical context for this article. 2016 was a year of RAPID growth for Uber. There was a running statistic internally that your employee ID would be at the median point just 6 months after being hired.

They were trying to hire (and poach) just about anyone they could around this time. Therefore, these articles are... very shiny, compared to the actual tech applied internally (note that even though Uber is referred to in the third person here, this is on uber.com and written by an Uber employee).

I worked at Uber for a year. Schemaless was... meh. Nobody really liked using it, nobody really understood it, and you weren't really allowed to host your own instance - you had to have another internal team do it for you, which didn't help the "understanding" problem.

It smelled distinctly of "not invented here" syndrome. A number of things inside Uber worked that way - the culture was so competitive and brutal, performance reviews were always a massacre, so everyone was trying to outshine their peers (or outright climb on their backs, etc).

This resulted in a LOT of "tech" being "invented" that 1:1 did something already prominent in open-source or was already an enterprise solution (probably cheaper than paying engineers to do it) but since actually achieving it and having your name on it meant you would look better for a promotion or a bonus or whatever over a colleague meant it was worth it to the individual to reinvent the wheel. Rinse and repeat over and over again.

I'm not an enemy of reinventing the wheel, mind you. But only if the new wheel works significantly better than the old one. This was rarely the case at Uber.

Postgres was still used somewhat commonly at Uber when I was there, but they were really pushing for Schemaless internally. It felt very overkill for just about everything outside the platform teams and was always, without fail, a massive pain to deal with.

Don't be fooled by these Uber engineering articles. This was PR to bolster up their OSS image to outsiders to help with hiring and poaching at the time. Things internally looked very different.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#84

Post by evanelias »

pizza234 wrote: post #15
frankietaylr wrote: post #10

Has Postgres architecture changed since Postgres 9.2 in terms of the inefficiencies mentioned in the article?

The main point, clustered vs. nonclustered indexing, is architectural, and not inherently inefficient; it depends on the use case. "Highly advanced" databases give both options, but AFAIK, MySQL/PGSQL will likely not offer this, at least for a very long time, since it requires radical changes.

On the one hand, MySQL has offered this for two decades, by virtue of pluggable storage engines being core to its design. Some storage engines use clustered indexes and some do not. The user can decide which one matches their use-case; very large companies can design their own custom special-purpose storage engines; etc.

On the other hand, mixing storage engines in a single db instance has operational downsides (especially re: crash-safe replication). And InnoDB is by far the dominant storage engine, and is probably unlikely to offer nonclustered indexing, so from that perspective I agree with your point.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#85

Post by Quekid5 »

amenonsen wrote: post #80
Quekid5 wrote: post #30

Earlier quoted context omitted.

I don't understand this. Why would it be sending queries and redoing on a replica and sending the WAL? Just sending the WAL would seem to be sufficient, or alternatively: sending queries would be redundant if you just send the WAL and apply directly at the secondaries.

It doesn't make sense to you because everything in the comment you're replying to is wrong. Neither log shipping (copying WAL files one by one) nor streaming replication (sending a stream of WAL) works by sending queries. WAL segments are 16MB by default, and the default archive_timeout is 0, not 1 minute (and the archive timeout is not applicable to streaming replication anyway). There is also nothing "adaptive" abo…

Thank you for confirming my suspicions :).

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#86

Post by [deleted] »

evanelias wrote: post #76
corty wrote: post #8

Earlier quoted context omitted.

PostgreSQL: "your date 2020-02-31 isn't a date, fix that" MySQL: "2020-02-31? Whatever man, I'll just enter something..."

MySQL's default settings reject invalid dates for over 5 years now, since MySQL 5.7. How long are people going to keep repeating this complaint? Literally every version of MySQL and MariaDB that allows invalid dates by default (MySQL 5.6 and older, MariaDB 10.1 and older) has reached end-of-life for upstream support from the vendor!

[deleted]

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#87

Post by midrus »

junon wrote: post #83

There's some historical context for this article. 2016 was a year of RAPID growth for Uber. There was a running statistic internally that your employee ID would be at the median point just 6 months after being hired. They were trying to hire (and poach) just about anyone they could around this time. Therefore, these articles are... very shiny, compared to the actual tech applied internally (note that even though Uber…

I think this applies to most companies. What they write in their blogs is a shiny, optimistic, limited view of the best part of their best system or similar. Once inside, things are never that great.

I myself was very ashamed of a company I worked for (also SF based) blog post... even the author of the post was a very well known open source maintainer of many libraries of a very popular programming language. Reading the posts in the blog was like.... I cannot believe we lie this big... internally things were just crap, and what the blog post made look like it was the norm, was just a side project of this person.

So, never trust companies blog posts by default.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#88

Post by petergeoghegan »

Tostino wrote: post #77
asah wrote: post #50

Earlier quoted context omitted.

Even though 99.9% of applications will never run into Uber's issues, it's been 4 years and 4 major versions later, and I'd love to review these complaints and see if they still apply to PG 13.

Wait until 14 for that comparison and it'll look much better. Bottom up index deletion helped solve some of the write amplification issues.

There is also index deduplication in Postgres 13 and the B-Tree enhancements in Postgres 12. All of these enhancements significantly improved the situation for workloads affected by what the blog post calls write amplification. (I myself call this phenomenon index version churn, since it is more descriptive and has less baggage.)

I was the author of all of the above, including the Postgres 14 work you mentioned (though Anastasia Lubennikova was the primary author of index deduplication). To me it feels like one very large project -- the effects are cumulative, and each major Postgres version had B-Tree work that built on the last release in one way or another.

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#89

Post by lumost »

TBH, everything that was listed as a complaint could be a complaint for nearly any transactional RDBMS. For workloads that require heavy always on replication and availability RDBMS's haven't been the go to solution for a long time vs. distributed DBs. Changing from Postgres to Mysql or MySQL to Postgres (or even Oracle) won't really buy you much if you're running into these issues.

Even this one

> The bug we ran into only affected certain releases of Postgres 9.2 and has been fixed for a long time now. However, we still find it worrisome that this class of bug can happen at all.

(rare/specific) Data Corruption bugs around master-promotion and handoff occur in every major DB. MySQL is no different, and I've personally had to track down issues in a few popular products. If you run thousands of copies of a piece of software with different workloads and hardware configurations... you're going to find bugs.

After all - how many DBs passed Jepsen on the first shot!

Re: Why Uber Engineering Switched from Postgres to MySQL (2016)

#90

Post by mlthoughts2018 »

midrus wrote: post #87
junon wrote: post #83

There's some historical context for this article. 2016 was a year of RAPID growth for Uber. There was a running statistic internally that your employee ID would be at the median point just 6 months after being hired. They were trying to hire (and poach) just about anyone they could around this time. Therefore, these articles are... very shiny, compared to the actual tech applied internally (note that even though Uber…

I think this applies to most companies. What they write in their blogs is a shiny, optimistic, limited view of the best part of their best system or similar. Once inside, things are never that great. I myself was very ashamed of a company I worked for (also SF based) blog post... even the author of the post was a very well known open source maintainer of many libraries of a very popular programming language. Reading…

I agree. I once worked for a company that wrote a blog post on reinventing and upgrading A/B tests, but the ugly truth was that the company couldn’t run an A/B test to save its life. Every A/B test was a disaster, there were fragmented different frameworks for A/B tests in different parts of the product, with inconsistent and unreliable data, and the core clickstream ingestion system that was the foundation for any possible way of testing would crash and go down for hours every few weeks, and product management would just silently ignore any effects of missing data or correlation between failures in different product features. Even though we had a team of statistical researchers, they were treated as if legit stats 101 concerns were just ivory tower academic hair splitting and often were silently omitted from being in A/B test design, recap or decision meetings, which were instead run by product managers with no stats training.

I remember interviewing a candidate once who said he was excited about the role because of that A/B testing blog post, and I just thought - geez what a completely soulless bait and switch ploy.

Post reply on HN