Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Uber Engineering Switched from Postgres to MySQL

eng.uber.com

Post reply on HN
Print view View original

241–250 of 306 posts

Re: Why Uber Engineering Switched from Postgres to MySQL

#241

Post by dialogbox »

pritambaral wrote: post #81
saurik wrote: post #53

Earlier quoted context omitted.

I would definitely not want PostgreSQL to "move to" the mechanism used by InnoDB as that mechanism is slower for reads (which this article even admits). This disk layout is one of the reasons I continue to use PostgreSQL. If you care more about write performance than reads you might even want to look into something entirely different than a traditional database. FWIW, PostgreSQL's mitigation to the write amplificatio…

Thank you for introducing me to "heap only tuples", I did not know about it. From what I read about it from Postgres's documentation[0], it would have helped in the updates where no indexed column was updated. 0: https://wiki.postgresql.org/wiki/Index-only_scans#Interactio...

You can find more detailed explanation from Bruce Momjian's site. (From page 64: https://momjian.us/main/writings/pgsql/mvcc.pdf)

Re: Why Uber Engineering Switched from Postgres to MySQL

#242

Post by wtbob »

codedokode wrote: post #80
rspeer wrote: post #63

Earlier quoted context omitted.

Except it's MySQL, so by the time you read it out it says: id first last birth_year 1 Blaise Pascal 1623 2 Gottfried Leibniz 1646 3 Emmy Noether 1882 4 Muhammad al-KhwÄ�rizmÄ« 780 5 Alan Turing 1912 6 Srinivasa Ramanujan 1887 7 Ada Lovelace 1815 8 Henri Poincaré 1854 (I know, I know. It is possible to configure MySQL encodings correctly. And given that they've put a lot of engineering thought into choosing MySQL, th…

I used both Postgres and MySQL and in my opinion encodings are more difficult to setup with Postgres. You need to change some "template" when creating a database to use unicode. Otherwise it will use latin1. I do not even understand what a database "template" is, how it is related to encodings, and why it is so overcomplicated. In MySQL there is no templates and you can change the encoding of a table at any time usin…

I can confirm. I recently set up a private mattermost server. Because the host I'm running it on didn't have the UTF-8 locale on by default (I think), Postgres defaulted to Latin-1, and mattermost ran fine, right up until someone sent a UTf-8 character which is unrepresentable in Latin-1.

That was No Fun™ to solve.

Re: Why Uber Engineering Switched from Postgres to MySQL

#243

Post by verma7 »

MikeKusold wrote: post #100
jobu wrote: post #91

Earlier quoted context omitted.

> It sounds like Uber is using MySQL as just a data bucket with primary keys They have a couple posts about "Schemaless", but I still don't understand why they used MySQL as the data store instead of something like Cassandra. ( https://eng.uber.com/schemaless-part-one/ ) From that post it looks like they basically built a no-sql database on top of a relational database. The only reason given was operational trust ( "…

I was at MesosCon and ended up talking to some Uber people. They are currently using Cassandra in prod. I can't speak as to why they use MySQL the way they do though.

I gave a talk at MesosCon about how we (are starting to) run Cassandra across multiple datacenters at Uber (https://www.youtube.com/watch?v=U2jFLx8NNro, https://schd.ws/hosted_files/mesosconna2016/60/mesoscon-uber...).

Re: Why Uber Engineering Switched from Postgres to MySQL

#245

Post by ksec »

pella wrote: post #58

Roadmaps ( PostgreSQL ) 2016-2017-... * Postgres Professional roadmap ( Pluggable storages, Multimaster cluster with sharding, Effective partitioning, Adaptive query planning, Page-level data compression, Connection pooling, Native querying for jsonb with indexing support, ....) https://wiki.postgresql.org/wiki/Postgres_Professional_roadm... * EnterpriseDB database server roadmap ( Parallelism, Replication, Vertical…

Why is there a Postgre Pro and Enterprise DB? Whats the differnce? ( Never heard of PostgrePro till today )

Re: Why Uber Engineering Switched from Postgres to MySQL

#246

Post by 0xmohit »

fdr wrote: post #92
0xmohit wrote: post #42

Worth quoting from the article: Accordingly, using pgbouncer to do connection pooling with Postgres has been generally successful for us. However, we have had occasional application bugs in our backend services that caused them to open more active connections (usually “idle in transaction” connections) than the services ought to be using, and these bugs have caused extended downtimes for us.

What I don't understand: no system gets to maintain open transactions for free. MySQL keeps UNDO logs, so the effect there, much like Oracle, is possibly running out of UNDO space, as well as slowing down reads that have to apply UNDO to get the last-committed row version. So what gives? Did Uber Engineering fix the dangling transaction issues while migrating off, or are they relying on some other property of MySQL v…

They seem to be relying on the database take care of those :)

Re: Why Uber Engineering Switched from Postgres to MySQL

#247

Post by dschiptsov »

Nice to see how Postgres (a relative of Informix) follows an old-school maxim to focus on Consistency and Durability by being "append-only" and never over-writing the data. Sticking to the right principles is better than over-optimization.

The Uber engineers should, perhaps, take a look at Changelogs of last 5 or so releases of MySQL to see how many bugs in InnoDB has been found in each release and read stories about data loses due to inability to repair storages.

According to old-school DBA tradition, it is much better to have an straightforward storage engine based on right principles written in C than fancy storage written in C++. At least if one values ones data.

Well, in the age of in-memory "databases", "durability through replication" and "eventual consistency" old school focus on disk commits might sound a bit outdated, until one gets that moment when shards got messed up and there is no way to know how many writes are missing and where.

Database is a durable storage which guarantees data consistency and ability to roll-back to a clean state through direct-access (by passing all caches) writes. At lest this is what we had in glorious times of IDS 7.3

Re: Why Uber Engineering Switched from Postgres to MySQL

#248

Post by philliphaydon »

Illniyar wrote: post #85
NhanH wrote: post #9

Well, this is heresy. Does that mean we are now officially boycotting Uber? Joke asides, one thing I've been trying to figure out for awhile is the limitation at which certain components/ systems broke down. Basically, something along the line of "given X records, this operations would take Y time, or would cause Z A B C problems". I've actually got developers friends asking me how fast a simple "SELECT * FROM X WHER…

"surprised that some NoSQL DB could do a query on hundred million rows in a few seconds" The tone seems to suggest this is fast, a simple index query in an RDBMS of even a hundred million rows will take milliseconds on even a weak computer.

I was testing some JSONB indexing and did a date-range query on a json doc for a table with 100m records to test with, it returned top 10 records that met the criteria in ~4s, figured out how to index the json doc for dates and this dropped to ~25ms.

This was running on an old Core i5 / 16gb of ram with a lot of junk running on Windows.

It would probably have been faster on a production environment.

Re: Why Uber Engineering Switched from Postgres to MySQL

#249

Post by philliphaydon »

pluma wrote: post #192
0xmohit wrote: post #5

Facebook maintains it's own fork [0] of MySQL. A couple of interesting talks are also available: MySQL at Facebook, Current and Future [1] and Massively Distributed Backup at Facebook Scale [2]. [0] https://github.com/facebook/mysql-5.6 [1] https://www.youtube.com/watch?v=jqwegP9xwVE [2] https://www.youtube.com/watch?v=UBHcmP2TSvk

Technically they use it as a dumb key-value storage.

There was a great talk from a FB engineer who talked about using MySQL as a key-value store, then another engineer posted a blog about how they use MySQL and queries and joins etc, I asked about the video and he got super defensive and said it was wrong and the guy had no idea what hes talking about.

It really put me off ever wanting to even consider working at Facebook.

Digging around I can't find either the blog or video :(

Re: Why Uber Engineering Switched from Postgres to MySQL

#250

Post by mercurial »

jlg23 wrote: post #233
mercurial wrote: post #165

Earlier quoted context omitted.

> The one huge advantage of an ORM is the ability to support multiple databases, but that only really works if you can do everything using the ORM. It has several advantages: support for multiple databases (which is useful, sometimes), the ability to serialize/deserialize an object graph in one go, and sometimes a decent query builder which lets you compose queries as opposed to concatenating strings. Unfortunately,…

> It has several advantages: [1] support for multiple databases (which is useful, sometimes), [2] the ability to serialize/deserialize an object graph in one go, and [3] sometimes a decent query builder which lets you compose queries as opposed to concatenating strings. [1] was the point of the comment you replied to and it provided a very important constraint as well [2] rarely needed & easy to implement with recurs…

I don't see how recursive queries are going to help you serialize a new Foo with a new field of type Bar with a new field of type FooBar, each going in a different table. That's what I mean by serializing an object graph.

As for building queries, SQL is straightforward (mostly). The problem is that it composes very badly. Any time you need to implement something like an advanced search (ie, lookup the same information, but with a number of different search criteria only known at runtime), the best you can do is concatenating partial queries and hoping you got the parentheses count right. Not to mention that a query builder will help with stuff the SQL syntax is miserable for, like IN clauses.

Post reply on HN