Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

111–120 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#111
post #87

Earlier quoted context omitted.

Postgres 17 tremendously improves vacuum performance

Vacuuming is a design decision that may have been valid back in the day, but is really a ball and chain today. In a low-resource environment deferring work makes sense. But even in low-resource environment the vacuum process would consume huge amounts of resources to do its job, especially given any kind of scale. And the longer it's deferred the longer the process will take. And if you actually are in a low-resource…

You run VACUUM often enough that you never need a VACUUM FULL. A normal VACUUM doesn't require any exclusive locks or a lot of disk space, so usually you can just run it in the background. Normally autovacuum does that for you, but at scale you transition to running it manually at low traffic times; or if you update rows a lot you throw more CPUs at the database server and run it frequently.

Vacuuming indices is a bit more finicky with locks, but you can just periodically build a new index and drop the old one when it becomes an issue

Re: Upgrading Uber's MySQL Fleet

#112
post #13

So satisfying to do a huge upgrade like this and then see the actual proof in the pudding with all the reduced latencies and query times.

Yeah some numbers caught my attention like ~94% reduction in overall database lock time. And to think they never have to worry about VACUUM. Ahh the peace.

MySQL indexes can contain references to rows in the undo log and has a periodic VACUUM-like process to remove those references, though no where near as impactful.

Re: Upgrading Uber's MySQL Fleet

#116
post #104

Earlier quoted context omitted.

As somebody who has always used MySQL, but always been told that I should be using Postgres, I'd love to understand what the issues with VACUUM are, and what I should be aware of when potentially switching databases?

Worth reading up on Postgres' MVCC model for concurrency.[0] Short version is that VACUUM is needed to clean up dead tuples and reclaim disk space. For most cases with smaller amounts of data, auto-vacuum works totally fine. But I've had issues with tables with 100m+ rows that are frequently updated where auto-vacuum falls behind and stops working completely. These necessitated a full data dump + restore (because we…

Thanks for that, that's valuable information.

Re: Upgrading Uber's MySQL Fleet

#117

File under "things you will never need to do if you use cloud services".

Nah random outages because the RDS instance you were on decided to faceplant itself, or the weird memory to bandwidth scaling AWS has chosen will make you pull your hair out on a high traffic day.

It's just different problems.

Re: Upgrading Uber's MySQL Fleet

#118
post #106
post #91

Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra r…

Didn't realize their entire MySQL data layer runs in AWS. Given that they went with basically a blue-green update strategy, this was, essentially a "witness our cloud spend" kind of post.

They're not. Almost all of their infra was on prem when I worked there 3 years ago.

Re: Upgrading Uber's MySQL Fleet

#120
post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong Don't know about your LLM feeling

I always thought 90% of what management wrote/said could be replaced by a RNN, and nowadays LLMs do even better!
Post reply on HN