Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

Post reply on HN
Print view View original

101–110 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#101

Post by pgwhalen »

sgarland wrote: post #57
edf13 wrote: post #5

3 million queries/second across 16k nodes seems pretty heavy on redundancy?

I was going to say, that's absolutely nothing. They state 2.1K clusters and 16K nodes; if you divide those, assuming even distribution, you get 7.6 instances/cluster. Round down because they probably rounded up for the article, so 1 primary and 6 replicas per cluster. That's still only ~1400 QPS / cluster, which isn't much at all. I'd be interested to hear if my assumptions were wrong, or if their schema and/or queri…

> assuming even distribution

I don't work for Uber, but this is almost certainly the assumption that is wrong. I doubt there is just a single workload duplicated 2.1K times. Additionally, different regions likely have different load.

Re: Upgrading Uber's MySQL Fleet

#102

Post by Jgrubb »

remon wrote: post #91

Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra r…

See, the problem is that the people who care about cost performance and the people who care about UX performance are rarely the same people, and often neither side is empowered with the data or experience they need to bridge the gap.

Re: Upgrading Uber's MySQL Fleet

#104

Post by mjr00 »

InsideOutSanta wrote: post #31
hu3 wrote: post #13

Earlier quoted context omitted.

Yeah some numbers caught my attention like ~94% reduction in overall database lock time. And to think they never have to worry about VACUUM. Ahh the peace.

As somebody who has always used MySQL, but always been told that I should be using Postgres, I'd love to understand what the issues with VACUUM are, and what I should be aware of when potentially switching databases?

Worth reading up on Postgres' MVCC model for concurrency.[0]

Short version is that VACUUM is needed to clean up dead tuples and reclaim disk space. For most cases with smaller amounts of data, auto-vacuum works totally fine. But I've had issues with tables with 100m+ rows that are frequently updated where auto-vacuum falls behind and stops working completely. These necessitated a full data dump + restore (because we didn't want to double our storage capacity to do a full vacuum). We fixed this by sharding the table and tweaking auto-vacuum to run more frequently, but this isn't stuff you have to worry about in MySQL.

Honestly if you're a small shop without database/postgres experts and MySQL performance is adequate for you, I wouldn't switch. Newer versions of MySQL have fixed the egregious issues, like silent data truncation on INSERT by default, and it's easier to maintain, in my experience.

[0] https://www.postgresql.org/docs/current/mvcc-intro.html

Re: Upgrading Uber's MySQL Fleet

#105

Post by greenie_beans »

robertlagrant wrote: post #97
greenie_beans wrote: post #71

Earlier quoted context omitted.

i don't know, i don't work there. i'm just somebody who almost died because one of their drivers was a terrible driver. that sounds like a problem they should figure out. dude didn't even know how to change a tire, so start with "basic knowledge of car maintenance." and a basic ability to speak english would be a good bar to meet, too. they'll let anybody with a driver's license, car, and a heart beat drive on that a…

If you have one big company with 10 bad drivers, you'll get a much worse impression of it than 100 companies each with one bad driver.

and your point is?

this just makes no sense bc the drivers are on all of the different apps. rework your formula.

Re: Upgrading Uber's MySQL Fleet

#106

Post by nunez »

remon wrote: post #91

Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra r…

Didn't realize their entire MySQL data layer runs in AWS. Given that they went with basically a blue-green update strategy, this was, essentially a "witness our cloud spend" kind of post.

Re: Upgrading Uber's MySQL Fleet

#107

Post by [deleted] »

remon wrote: post #91

Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra r…

[deleted]

Re: Upgrading Uber's MySQL Fleet

#108

Post by abhorrence »

blindriver wrote: post #90
paulryanrogers wrote: post #21

Earlier quoted context omitted.

Considering several versions of 8.0 had a crashing bug if you renamed a table, waiting is probably the right choice.

You’re not renaming tables when you’re at scale.

Sure you do! It's how online schema changes tend to be done, e.g. https://docs.percona.com/percona-toolkit/pt-online-schema-ch... describes doing an atomic rename as the last step.

Re: Upgrading Uber's MySQL Fleet

#110

Post by msoad »

remon wrote: post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

Yeah, I kinda stopped reading when I felt this. Not sure why? The substance is still interesting and worth learning from but knowing LLM wrote it made me feel icky a little bit
Post reply on HN