Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

Post reply on HN
Print view View original

211–215 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#211

Post by blindriver »

paulryanrogers wrote: post #196
yen223 wrote: post #170

Earlier quoted context omitted.

You aren't renaming tables at scale because there are 27 downstream services that will break if you even think about fixing the name of the revnue_dolars table, and it's not in anyone's OKR to fix it

Take a closer look at how some min downtime tools work. They often use a drop-swap strategy to replace an old table name with a new one that has schema changes. There are sometimes temporary views to keep the old and new code working during gradual transitions.

You clearly have not worked at global companies with concerns that go beyond “can I do it?”. I’ve used percona and I’ve used these tools before and percona is great. But when you’re at scale something as trivial as a table name change is not something you ever do because it’s not worth the risk of taking your entire site down. Just use the existing name no matter how bad it is.

Re: Upgrading Uber's MySQL Fleet

#212

Post by paulryanrogers »

blindriver wrote: post #211
paulryanrogers wrote: post #196

Earlier quoted context omitted.

Take a closer look at how some min downtime tools work. They often use a drop-swap strategy to replace an old table name with a new one that has schema changes. There are sometimes temporary views to keep the old and new code working during gradual transitions.

You clearly have not worked at global companies with concerns that go beyond “can I do it?”. I’ve used percona and I’ve used these tools before and percona is great. But when you’re at scale something as trivial as a table name change is not something you ever do because it’s not worth the risk of taking your entire site down. Just use the existing name no matter how bad it is.

Are we talking about the same Pt-online-ddl-change?

It certainly has a drop-swap approach which does a table rename, even though the resulting table ends up with the same name as before.

That's how I encountered the MySQL table-rename-crash bug.

Re: Upgrading Uber's MySQL Fleet

#213

Post by pizza234 »

hu3 wrote: post #168
pizza234 wrote: post #14

Earlier quoted context omitted.

I suppose they opted for a conservative upgrade policy, as v8.4 probably includes all the functional additions/changes of the previous v8.1+ versions, and moving to it would have been a very big step. MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.

> MySQL is very unstable software I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly. As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers Unless you know something we don't and we're just lucky.

I've used MySQL for almost 15 years, and reported dozens of bugs, a few of which were critical security issues.

My definition of "very unstable" has been very vague, so I'll clarify.

MySQL is stable software if a company uses a subset of features - the core ones, stable for a long time. Anything outside of that area, and either a company has very considerable engineering resources (that is, being able to work on the MySQL source, indeed like Uber, Google or Facebook), or it's suicide.

For starters, the MySQL development model has a habit of abandoning functionalities while not officially deprecating them, leaving them buggy.

MyISAM for example has been abandoned long ago, but not officially. Around 10/12 years ago, while it was still somewhat used, we experienced a bug where the tables were marked as crashed on clean shutdowns. Very nasty because it caused long recovery times on startup.

InnoDB fulltext indexes are the same. They have a broken workflow (they require maintenance, but some required maintenance functionalities are missing), and clearly MySQL has abandoned them, but not officially deprecated.

New functionalities have typically nasty bugs. The critical bugs I've mentioned were on simple operations related to relatively new, but not widespread, functionalities (MVIs, regular expressions, InnoDB FTIs, I think JSON, and can't remember the other(s)).

Worse, MySQL 8.0 has been a complete trainwreck, because the product managers decided to develop features/changes in patch versions, continuosly introducing breakages. So, maybe a functionality that has finally been stable for an year or more, suddenly breaks (in production, of course).

Heck, even something as simple as 'SHOW TRIGGERS' is currently buggy, just because triggers are not common in the userbase.

Given the domain you've mentioned (banking) you've probably been working with a conservative set of functionalities - which absolutely makes sense, but calling MySQL stable depends on whether one considers MySQL a small set of functionalities, or the whole set of them.

Re: Upgrading Uber's MySQL Fleet

#214

Post by anitil »

I wonder why they did a large version jump in one shot (v5.7->v8) and didn't do incremental upgrades (v5.7 -> 6.x etc)?

I wonder because the promotion of the secondary v8 node to primary is a breaking change in this path, whereas in an incremental upgrade it might not have been. But I also understand at this sort of scale things might be as easy as that.

Post reply on HN