Live data from Hacker News

Why we lost Uber as a user

postgresql.org

301–310 of 310 posts

Re: Why we lost Uber as a user

#301
post #212

Earlier quoted context omitted.

Absolutely - going from Postgres to MySQL is only trading one set of problems for another. It's a longer runway, but not infinite. Their exact use-case is what things like Cassandra were built for - insanely high writes / updates. They're also built to split your load across N systems, as long as you're still using a monolithic database (even with read replicas) you physically can't get the same performance that you…

An Uber engineer at a conference said that none of the open-source NoSQL systems could handle their load, and they they had to heavily hack one of them (which I think was Cassandra but the memory is vague) to get the last bit of performance out of it while they were building Schemaless.

They seem to be running Cassandra still, at least as of last month - "Running Cassandra on Apache Mesos Across Multiple Datacenters at Uber" https://www.youtube.com/watch?v=U2jFLx8NNro

Re: Why we lost Uber as a user

#303

Earlier quoted context omitted.

You can't configure MySQL to not do "any" of that. You can certainly make it better, but there simply aren't options to configure away all of the boneheadedness. There are also tons of hidden gotchas that exist in, for example, the query planner. It can be extremely fickle and suddenly switch from a performant query plan to a terrible one that creates unindexed temporary tables and sorts them or joins against them. O…

You're thinking there's databases out there that are flawless, that never corrupt data, but that's garbage. They all do to a degree. They're also subject to being corrupted by hardware failures that aren't related to software. Anyone with a huge production database running under load is going to have ways of mitigating these problems. Tumblr manages with MySQL, they open-sourced some of their tools like JetPants ( ht…

Nothing in my comment implies anything close to thinking that there are databases that are don't or flawless or don't corrupt data. I'm not sure how a comment about poor query optimization could possibly be interpreted that way.

Regardless, MySQL by default silently eats data in common situations (truncation of VARCHAR) and returns flat-out incorrect results due to PHP-style "helpful" coercions (SELECT 0 == "banana"). It implements UTF-8 incorrectly, but fixing it would break existing apps, so we're forever stuck with "utf8" encoding that isn't.

There are a million more of these, and while some of them have workarounds (strict tables, utf8mb4), many of them don't (automatic coercion, boneheaded query planner, creating implicit temporary tables without indexes even when present, etc.).

A comparison of MySQL to PHP is apt, honestly. The fact that PHP is a blight doesn't mean other languages don't have their own problems. But PHP (like MySQL) is in a league of its own here.

Re: Why we lost Uber as a user

#304
post #211

Earlier quoted context omitted.

I can confirm that the query optimizer introduced a rather serious bug (significantly suboptimal plan for queries involving low cardinality indices), which caused serious issues in our system. This, in addition to the fact that index merging has been broken in MySQL 5.6 for more than an year now (in some cases it will cause empty resultsets to be returned), and that it is still broken on MySQL 5.7

Do you happen to have a link to the bug report for the first issue you described? I'm wondering whether I saw a similar thing in a benchmark I tested.

Sorry, don't have it. We experienced three separate instances of it, and have to open a report yet.

Re: Why we lost Uber as a user

#305
post #258

Earlier quoted context omitted.

You're not wrong. But mostly we haven't collectively agreed that relational databases aren't great for highly-indexed rapid-update join tables. I think we will at some point. That's the primary original use case for a lot of NoSQL, and the reason Twitter had so much trouble with relational databases. But these are cultural understandings, and those move slowly. Also, we're poorly (collectively) equipped to handle sub…

How has NoSQL addressed join tables? All the approaches I've seen are much, much slower than with a traditionally vertically scaled relational database—or by moving away from joining altogether—it's traditionally been the twin pressures of scale and replication that force people to move to a distributed database.

Cassandra, for instance, makes it possible to query your join-type tables in relatively manual chunks using slice queries. You have to do a lot more of it in the application so you wind up with weird partial failures...

But at scale, weird partial failures, results as they are found and eventual consistency are usually preferable to a really long DB query that never returns.

Above a certain scale, big joins are simply not usable. NoSQL's manage-it-yourself approach is good for getting partial results where full results are too expensive.

You can think of it as applying a heuristic approach where an exact approach is too expensive, if it makes you feel less like NoSQL sullies the purity of databases :-)

Re: Why we lost Uber as a user

#306
post #219
post #17

Earlier quoted context omitted.

Just curious, what's your definition of "Tech Company"? All services provided by Uber are purely technical. Drivers and Riders are customers of Uber's technology. The full name of the company is "Uber Technologies Inc."

I wasn't sure what to think of that comment either when I first read it, but I sort of see where he is coming from. One side you have companies like Oracle, Microsoft, and IBM types that actually develop new forms of technology and sell the technology to people. Then there are companies that leverage technology in other industries to "disrupt" like OpenTable, Uber, and AirBnB. Then there are companies like Google and…

Interesting perspective, but I don't think selling technology directly should be the requirement. I think a company is a "Tech Company" when the core product is technology produced by the company. I consider Facebook and Twitter to be tech companies, even though they don't sell technology directly. AirBnB and Uber have some decent open source contributions and have the ability to contribute back huge advances but probably won't focus as much on that until they're profitable.

Re: Why we lost Uber as a user

#307
post #258

Earlier quoted context omitted.

You're not wrong. But mostly we haven't collectively agreed that relational databases aren't great for highly-indexed rapid-update join tables. I think we will at some point. That's the primary original use case for a lot of NoSQL, and the reason Twitter had so much trouble with relational databases. But these are cultural understandings, and those move slowly. Also, we're poorly (collectively) equipped to handle sub…

How has NoSQL addressed join tables? All the approaches I've seen are much, much slower than with a traditionally vertically scaled relational database—or by moving away from joining altogether—it's traditionally been the twin pressures of scale and replication that force people to move to a distributed database.

Non-relational DBs (the Not Only SQL sort) usually passes the buck up to the application layer for several things. It is in a way a good thing to do if the DB wants to focus on scale out issues such as consistency of replicas or high availability. I personally find it pretty easy to work with alternate data models, thus not relying on the possibility of a join or a transaction, as long as I am not writing a banking application. Without a relational data model, the biggest problem becomes verification IMHO; verification of the DB, the application logic, and whether the application logic makes the right assumptions about the DB it is using.

Re: Why we lost Uber as a user

#308
The whole discussion (this and [1] and [2]) is very interesting. I find it that Uber is holding on the ACID guarantees of a relational DB so much, in a way that is clearly hurting them. IMHO it will do them a big service to break down the responsibility of such a DB into multiple distributed systems that can work on a global scale. For example, a distributed lock system can help them when they need transactions. If they keep moving from one relational DB to another, they are bound to hit problems of Availability because they are choosing very strong Consistency, and the CAP theorem says we can't have it all (Partition tolerance is required).

[1] https://news.ycombinator.com/item?id=12166585 [2] https://news.ycombinator.com/item?id=12179222

Re: Why we lost Uber as a user

#309

Earlier quoted context omitted.

Among other things, there's a substantial performance penalty for secondary index lookups with clustered indices (since they need to traverse two index structures).

I'm not sure about the other databases, but in MS SQL you can include columns in the index leaves. This mitigates the need for the second lookup and can be even faster than the Postgres approach.

Keep in mind that the complaint Uber had was with writes in Postgres affecting secondary indices that didn't cover a particular column. As far as I know, in all databases, if a covered column is modified the secondary index must be, too. So that does not really resolve Uber's problem.

Additionally, I can't quite recall whether this is the case in MS SQL (since it uses pessimistic locking by default, not snapshot isolation, which has different performance characteristics), but in most MVCC architectures there's the additional problem that the underlying row value could have been changed concurrently with your query (e.g., deleted or modified). While this might not seem so bad for simple row-level lookups, this gets much more problematic if you are doing something like a range query, where the index might contain rows that weren't in the database at the beginning of your snapshot. There are a variety of ways of dealing with that problem, but most of them involve increased write traffic (index sizes get even more bloated because now they need versioning information), increased read traffic (index reads that touch out-of-date rows may have to follow an undo pointer, requiring the extra seeks you were trying to avoid in the first place), more locking (for instance, you could lock the entire index when a transaction modified it to keep it consistent--but that would decrease concurrency--or you could try to do range locking--which can lead to increased probability of deadlocks and also decreases concurrency and increases contention on the lock manager), opportunistic optimizations that only work on mostly-immutable data (Postgres and HyPer's solutions is to maintain a much smaller visibility map with bits indicating whether it's safe to assume the index is unchanged), or giving up multi-key read consistency (I'm assuming if this were an option you would be using another storage engine, because lots of them can perform unbelievably well if that restriction is relaxed!).

My point being, there's no such thing as a free lunch. Personally, I'm usually extremely happy to give up on pessimistic locking for the concurrency benefits of MVCC, and index utility decreases sharply as it gets larger, but as with many other things it entirely depends on your workload. Two-phase locking actually works far better than MVCC of any sort under heavy contention with short transactions (what Uber is apparently doing), so they really probably should have investigated SQL Server or another database optimized for pessimistic concurrency control.

Re: Why we lost Uber as a user

#310
post #119

Earlier quoted context omitted.

Google's hosted CloudSQL (a hosted &customized MySQL solution) had tons of connection drop issues. Last incident it took them 72+hrs to resolve and they had no idea what and why caused it (the resolution was a side effect from another issue that resolved for some other custom complains.)

That was 1st gen CloudSQL right? 2nd Gen CloudSQL seems much more similar to running your own MySQL server. (Both the pros and cons of that)

Those were 2nd Gen CloudSQL instances. :(

We jumped onto the 2nd gen instances even when they were in beta.

Post reply on HN