Earlier quoted context omitted.
Absolutely - going from Postgres to MySQL is only trading one set of problems for another. It's a longer runway, but not infinite. Their exact use-case is what things like Cassandra were built for - insanely high writes / updates. They're also built to split your load across N systems, as long as you're still using a monolithic database (even with read replicas) you physically can't get the same performance that you…
An Uber engineer at a conference said that none of the open-source NoSQL systems could handle their load, and they they had to heavily hack one of them (which I think was Cassandra but the memory is vague) to get the last bit of performance out of it while they were building Schemaless.
Why we lost Uber as a user
301–310 of 310 posts
Re: Why we lost Uber as a user
#302Re: Why we lost Uber as a user
#303Earlier quoted context omitted.
You can't configure MySQL to not do "any" of that. You can certainly make it better, but there simply aren't options to configure away all of the boneheadedness. There are also tons of hidden gotchas that exist in, for example, the query planner. It can be extremely fickle and suddenly switch from a performant query plan to a terrible one that creates unindexed temporary tables and sorts them or joins against them. O…
You're thinking there's databases out there that are flawless, that never corrupt data, but that's garbage. They all do to a degree. They're also subject to being corrupted by hardware failures that aren't related to software. Anyone with a huge production database running under load is going to have ways of mitigating these problems. Tumblr manages with MySQL, they open-sourced some of their tools like JetPants ( ht…
Regardless, MySQL by default silently eats data in common situations (truncation of VARCHAR) and returns flat-out incorrect results due to PHP-style "helpful" coercions (SELECT 0 == "banana"). It implements UTF-8 incorrectly, but fixing it would break existing apps, so we're forever stuck with "utf8" encoding that isn't.
There are a million more of these, and while some of them have workarounds (strict tables, utf8mb4), many of them don't (automatic coercion, boneheaded query planner, creating implicit temporary tables without indexes even when present, etc.).
A comparison of MySQL to PHP is apt, honestly. The fact that PHP is a blight doesn't mean other languages don't have their own problems. But PHP (like MySQL) is in a league of its own here.
Re: Why we lost Uber as a user
#304Earlier quoted context omitted.
I can confirm that the query optimizer introduced a rather serious bug (significantly suboptimal plan for queries involving low cardinality indices), which caused serious issues in our system. This, in addition to the fact that index merging has been broken in MySQL 5.6 for more than an year now (in some cases it will cause empty resultsets to be returned), and that it is still broken on MySQL 5.7
Do you happen to have a link to the bug report for the first issue you described? I'm wondering whether I saw a similar thing in a benchmark I tested.
Re: Why we lost Uber as a user
#305Earlier quoted context omitted.
You're not wrong. But mostly we haven't collectively agreed that relational databases aren't great for highly-indexed rapid-update join tables. I think we will at some point. That's the primary original use case for a lot of NoSQL, and the reason Twitter had so much trouble with relational databases. But these are cultural understandings, and those move slowly. Also, we're poorly (collectively) equipped to handle sub…
How has NoSQL addressed join tables? All the approaches I've seen are much, much slower than with a traditionally vertically scaled relational database—or by moving away from joining altogether—it's traditionally been the twin pressures of scale and replication that force people to move to a distributed database.
But at scale, weird partial failures, results as they are found and eventual consistency are usually preferable to a really long DB query that never returns.
Above a certain scale, big joins are simply not usable. NoSQL's manage-it-yourself approach is good for getting partial results where full results are too expensive.
You can think of it as applying a heuristic approach where an exact approach is too expensive, if it makes you feel less like NoSQL sullies the purity of databases :-)
Re: Why we lost Uber as a user
#306Earlier quoted context omitted.
Just curious, what's your definition of "Tech Company"? All services provided by Uber are purely technical. Drivers and Riders are customers of Uber's technology. The full name of the company is "Uber Technologies Inc."
I wasn't sure what to think of that comment either when I first read it, but I sort of see where he is coming from. One side you have companies like Oracle, Microsoft, and IBM types that actually develop new forms of technology and sell the technology to people. Then there are companies that leverage technology in other industries to "disrupt" like OpenTable, Uber, and AirBnB. Then there are companies like Google and…
Re: Why we lost Uber as a user
#307Earlier quoted context omitted.
You're not wrong. But mostly we haven't collectively agreed that relational databases aren't great for highly-indexed rapid-update join tables. I think we will at some point. That's the primary original use case for a lot of NoSQL, and the reason Twitter had so much trouble with relational databases. But these are cultural understandings, and those move slowly. Also, we're poorly (collectively) equipped to handle sub…
How has NoSQL addressed join tables? All the approaches I've seen are much, much slower than with a traditionally vertically scaled relational database—or by moving away from joining altogether—it's traditionally been the twin pressures of scale and replication that force people to move to a distributed database.
Re: Why we lost Uber as a user
#308[1] https://news.ycombinator.com/item?id=12166585 [2] https://news.ycombinator.com/item?id=12179222
Re: Why we lost Uber as a user
#309Earlier quoted context omitted.
Among other things, there's a substantial performance penalty for secondary index lookups with clustered indices (since they need to traverse two index structures).
I'm not sure about the other databases, but in MS SQL you can include columns in the index leaves. This mitigates the need for the second lookup and can be even faster than the Postgres approach.
Additionally, I can't quite recall whether this is the case in MS SQL (since it uses pessimistic locking by default, not snapshot isolation, which has different performance characteristics), but in most MVCC architectures there's the additional problem that the underlying row value could have been changed concurrently with your query (e.g., deleted or modified). While this might not seem so bad for simple row-level lookups, this gets much more problematic if you are doing something like a range query, where the index might contain rows that weren't in the database at the beginning of your snapshot. There are a variety of ways of dealing with that problem, but most of them involve increased write traffic (index sizes get even more bloated because now they need versioning information), increased read traffic (index reads that touch out-of-date rows may have to follow an undo pointer, requiring the extra seeks you were trying to avoid in the first place), more locking (for instance, you could lock the entire index when a transaction modified it to keep it consistent--but that would decrease concurrency--or you could try to do range locking--which can lead to increased probability of deadlocks and also decreases concurrency and increases contention on the lock manager), opportunistic optimizations that only work on mostly-immutable data (Postgres and HyPer's solutions is to maintain a much smaller visibility map with bits indicating whether it's safe to assume the index is unchanged), or giving up multi-key read consistency (I'm assuming if this were an option you would be using another storage engine, because lots of them can perform unbelievably well if that restriction is relaxed!).
My point being, there's no such thing as a free lunch. Personally, I'm usually extremely happy to give up on pessimistic locking for the concurrency benefits of MVCC, and index utility decreases sharply as it gets larger, but as with many other things it entirely depends on your workload. Two-phase locking actually works far better than MVCC of any sort under heavy contention with short transactions (what Uber is apparently doing), so they really probably should have investigated SQL Server or another database optimized for pessimistic concurrency control.
Re: Why we lost Uber as a user
#310Earlier quoted context omitted.
Google's hosted CloudSQL (a hosted &customized MySQL solution) had tons of connection drop issues. Last incident it took them 72+hrs to resolve and they had no idea what and why caused it (the resolution was a side effect from another issue that resolved for some other custom complains.)
That was 1st gen CloudSQL right? 2nd Gen CloudSQL seems much more similar to running your own MySQL server. (Both the pros and cons of that)
We jumped onto the 2nd gen instances even when they were in beta.