Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Uber Engineering Switched from Postgres to MySQL

eng.uber.com

Post reply on HN
Print view View original

91–100 of 306 posts

Re: Why Uber Engineering Switched from Postgres to MySQL

#91

Post by jobu »

ledjon wrote: post #51

I would argue that most of these Postgres "flaws" are actually advantages over MySQL when you look at them holistically rather than the very specific Uber use-case. Postgres's MVCC is superior (can rollback DDL, can add indexes online, can have open read transactions for a VERY long time without impacting other parts of the system) Postgres supports many types of indexes, not just b-tree. One thing it doesn't have is…

> It sounds like Uber is using MySQL as just a data bucket with primary keys

They have a couple posts about "Schemaless", but I still don't understand why they used MySQL as the data store instead of something like Cassandra. ( https://eng.uber.com/schemaless-part-one/ ) From that post it looks like they basically built a no-sql database on top of a relational database.

The only reason given was operational trust ( "If we get paged at 3 am when the datastore is not answering queries and takes down the business, would we have the operational knowledge to quickly fix it?" ). The project took nearly a year to roll out, and in that time the operation knowledge could surely be trained, hired, or contracted.

Re: Why Uber Engineering Switched from Postgres to MySQL

#92

Post by fdr »

0xmohit wrote: post #42

Worth quoting from the article: Accordingly, using pgbouncer to do connection pooling with Postgres has been generally successful for us. However, we have had occasional application bugs in our backend services that caused them to open more active connections (usually “idle in transaction” connections) than the services ought to be using, and these bugs have caused extended downtimes for us.

What I don't understand: no system gets to maintain open transactions for free. MySQL keeps UNDO logs, so the effect there, much like Oracle, is possibly running out of UNDO space, as well as slowing down reads that have to apply UNDO to get the last-committed row version. So what gives? Did Uber Engineering fix the dangling transaction issues while migrating off, or are they relying on some other property of MySQL vs. Postgres?

Re: Why Uber Engineering Switched from Postgres to MySQL

#93

Post by pritambaral »

xrstf wrote: post #46
swasheck wrote: post #3

this reads like a laundry list of buzzwords that were designed to justify not throwing any effort into postgresql and just going with a new shiny toy (not mysql. yes. i know it's been around for a while). it happens everywhere.

> not throwing any effort into postgresql The amount of research about the on-disk internals of both PostgreSQL and MySQL are a lot more effort than I would have probably spent (granted, I don't have a team of highly paid devs at my disposal, but still, I've seen technical decisions made on the basis of Google Trends...).

In my personal opinion, those bits of research aren't much, and they would have found it when they looked around for the cause of the write-amplification. A truly good amount of research would also have popped up heap-only-tuples, as mentioned here[0], or other ways to mitigate too many secondary-index updates.

0: https://news.ycombinator.com/item?id=12167161

Re: Why Uber Engineering Switched from Postgres to MySQL

#94

Post by rosser »

codedokode wrote: post #80
rspeer wrote: post #63

Earlier quoted context omitted.

Except it's MySQL, so by the time you read it out it says: id first last birth_year 1 Blaise Pascal 1623 2 Gottfried Leibniz 1646 3 Emmy Noether 1882 4 Muhammad al-KhwÄ�rizmÄ« 780 5 Alan Turing 1912 6 Srinivasa Ramanujan 1887 7 Ada Lovelace 1815 8 Henri Poincaré 1854 (I know, I know. It is possible to configure MySQL encodings correctly. And given that they've put a lot of engineering thought into choosing MySQL, th…

I used both Postgres and MySQL and in my opinion encodings are more difficult to setup with Postgres. You need to change some "template" when creating a database to use unicode. Otherwise it will use latin1. I do not even understand what a database "template" is, how it is related to encodings, and why it is so overcomplicated. In MySQL there is no templates and you can change the encoding of a table at any time usin…

So ... you'd rather have a database with mixed encodings, on a table-by-table basis?

Ugh.

Re: Why Uber Engineering Switched from Postgres to MySQL

#96

Post by stephengillie »

GuiA wrote: post #55
stephengillie wrote: post #44

Earlier quoted context omitted.

Almost all problems are already solved. "Solving" a problem today is mostly ego-stroking.

Sweet, I have a few PhD thesis and NSF grant proposals I'd like your help on.

I'd love to help stroke your ego, but I see you're busy doing it yourself.

Re: Why Uber Engineering Switched from Postgres to MySQL

#97

Post by busterarm »

tracker1 wrote: post #66
millamox wrote: post #30

Earlier quoted context omitted.

I strongly disagree. it would have been useful f they'd stuck to problems without well-known solutions. Sadly, they also mixed in issues which are easily solved, or in a particularly egregious case, where they just complain about a bug. As though MySQL never had a bug. That was silly. My read of it was: Postgres annoyed us a few times, and we got fed up with its, so now something different will annoy us. Please look…

I got a similar impression... though with Uber's scale, funding and resources, they probably could have worked with and through their issues with Postgres. I'm actually surprised they didn't take a multi-pronged approach to their issues. Since they're using Schemaless, I'm curious why they didn't go for one of the many non-sql databases that may well be a much closer match to their use case. It seems to me that Cassa…

I'll preface what I'm about to say with "I have never worked for Uber and I don't know terribly much about their internal structure", but from my interviews with Uber and a few of their hires I know, it seems that they tend towards hiring totally independent teams from the existing staff when tackling big projects...including hiring an outsider manager to hire a whole team. I won't speculate as to the reasons for this publicly but I've drawn some interesting conclusions from this.

A multi-pronged approach that might involve multiple stakeholders just doesn't seem like their way of doing things.

Re: Why Uber Engineering Switched from Postgres to MySQL

#98

Post by tracker1 »

brianwawok wrote: post #61
vbezhenar wrote: post #48

Why would anyone run hundreds of connections? A server can only process number_of_processor_cores connections at once. Sure, few connections might wait for I/O, but not hundreds, unless database is very untypical.

For example: You are using Python and you have 10 web servers and 20 background servers connected to a common DB. Each server has 10 threads, and each thread holds 1 connection open. That is 300 open connections. Opening and closing connections is very slow and expensive, so almost always better to keep these 300 connections open than to try to be fancy. You COULD try to say give each server only 3 connections and ma…

Agreed, which is why many clients create a pool of connections that gets reused. Connection cost is expensive, and the rdbms already handles concurrency and even a couple thousand connections shouldn't be a significant overhead.

Re: Why Uber Engineering Switched from Postgres to MySQL

#99

Post by madhusudhan000 »

raisyer wrote: post #20

while Postgress might be better if you use it 'as-is'.... community of MySQL is much better and the tools available are more mature... just goes on to prove that even if something is not-that-good.. it still might be successful,scalable and popular if there is a strong community behind it..

>community of MySQL is much better and the tools available are more mature

Can you backup your statements with some facts. I have seen the postgres community to be much better particularly now with Oracle taking ownership.

Re: Why Uber Engineering Switched from Postgres to MySQL

#100

Post by MikeKusold »

jobu wrote: post #91
ledjon wrote: post #51

I would argue that most of these Postgres "flaws" are actually advantages over MySQL when you look at them holistically rather than the very specific Uber use-case. Postgres's MVCC is superior (can rollback DDL, can add indexes online, can have open read transactions for a VERY long time without impacting other parts of the system) Postgres supports many types of indexes, not just b-tree. One thing it doesn't have is…

> It sounds like Uber is using MySQL as just a data bucket with primary keys They have a couple posts about "Schemaless", but I still don't understand why they used MySQL as the data store instead of something like Cassandra. ( https://eng.uber.com/schemaless-part-one/ ) From that post it looks like they basically built a no-sql database on top of a relational database. The only reason given was operational trust ( "…

I was at MesosCon and ended up talking to some Uber people. They are currently using Cassandra in prod. I can't speak as to why they use MySQL the way they do though.
Post reply on HN