Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

Post reply on HN
Print view View original

191–200 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#191

Post by sroussey »

yen223 wrote: post #170
abhorrence wrote: post #108

Earlier quoted context omitted.

Sure you do! It's how online schema changes tend to be done, e.g. https://docs.percona.com/percona-toolkit/pt-online-schema-ch... describes doing an atomic rename as the last step.

You aren't renaming tables at scale because there are 27 downstream services that will break if you even think about fixing the name of the revnue_dolars table, and it's not in anyone's OKR to fix it

I’m not sure if this is a joke (ref to OKR was funny, for example), or just naive and not understanding the parent comment. I found it funny either way though.

Re: Upgrading Uber's MySQL Fleet

#192

Post by calmoo »

antisthenes wrote: post #70
calmoo wrote: post #41

Earlier quoted context omitted.

There's nothing that sticks out to me as obviously Indian English in this blog post. It's almost certainly entirely run through an LLM though.

If there are large amounts of Indian English in an LLM's training data, it stands to reason the LLM output will be very similar to Indian English, no?

No, I don't think so. There's nothing Indian English about the blog post. It's just overly verbose, fluffy language.

Re: Upgrading Uber's MySQL Fleet

#193

Post by jcgl »

BowBun wrote: post #153
xyst wrote: post #86

I wonder if an upgrade like this would be less painful if the db layer was containerized? The migration process they described would be less painful with k8s. Especially with 2100+ nodes/VMs

A pipe dream. Having recently interacted with a modern k8s operator for Postgres, it lacked support for many features that had been around for a long time. I'd be surprised if MySQL's operators are that much better. Also consider the data layer, which is going to need to be solved regardless. Of course at Uber's scale they could write their own, I guess. At that point, if you're reaching in and scripting your pods to…

> it lacked support for many features that had been around for a long time

Care to elaborate at all? Were they more like missing edge cases or absent core functionality? Not to imply that missing edge cases aren’t important when it comes to DB ops.

Re: Upgrading Uber's MySQL Fleet

#194

Post by Groxx »

wongarsu wrote: post #145
est31 wrote: post #123

Earlier quoted context omitted.

It contains the word "delve", a word that got way more popular in use since the introduction of LLMs. Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive: We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before emba…

It doesn't get much better if you translate that paragraph from corpo speak to normal language: "We did the upgrade step by step. We did each step in batches. After we already decided how we were going to upgrade the clusters but before actually doing it we asked the teams responsible for keeping the clusters running for their opinion. This helped create an environment where we work together and helped monitoring the…

People write like that to sound good to higher-ups who don't understand what's going on underneath.

There's A LOT of that kind of content to learn from. A brief glance at LinkedIn is all you need.

Re: Upgrading Uber's MySQL Fleet

#196

Post by paulryanrogers »

yen223 wrote: post #170
abhorrence wrote: post #108

Earlier quoted context omitted.

Sure you do! It's how online schema changes tend to be done, e.g. https://docs.percona.com/percona-toolkit/pt-online-schema-ch... describes doing an atomic rename as the last step.

You aren't renaming tables at scale because there are 27 downstream services that will break if you even think about fixing the name of the revnue_dolars table, and it's not in anyone's OKR to fix it

Take a closer look at how some min downtime tools work. They often use a drop-swap strategy to replace an old table name with a new one that has schema changes.

There are sometimes temporary views to keep the old and new code working during gradual transitions.

Re: Upgrading Uber's MySQL Fleet

#197

Post by sgarland »

anonzzzies wrote: post #188
sgarland wrote: post #147

Earlier quoted context omitted.

People not realizing you can tune autovacuum on a per-table basis is the big one. Autovacuum can get a lot done if you have enough workers and enough spare RAM to throw at them. For indices, as you mentioned, doing either a REINDEX CONCURRENTLY (requires >= PG12), or a INDEX CONCURRENTLY / DROP CONCURRENTLY (and a rename if you’d like) is the way to go. In general, there is a lot more manual maintenance needed to kee…

We make good money 'saving' people from Aurora; you can throw traffic at it and pay more. We often migrate companies who then end up with a fraction of the price.

I’m convinced that Aurora’s team consists mostly of sales. There are certainly some talented engineers working on it – I’ve talked to a few – but by and large, all of my interactions with AWS about DB stuff was been them telling me how much better it is than other options.

I’ve tested Aurora Postgres and MySQL against both RDS and native (on my own, extremely old hardware), and Aurora has never won in performance. I’ve been told that “it’s better in high concurrency,” but IMO, that’s what connection poolers are for.

Re: Upgrading Uber's MySQL Fleet

#198

Post by sgarland »

notyourwork wrote: post #185
JackSlateur wrote: post #182

Earlier quoted context omitted.

A couple of years ago, I optimize some shit and reduced the annual billing of 150k€/y, for a 3 days of work I might say, "hardware" is expensive compared to (my) salary :)

There isn’t always low hanging fruit. And when there is, it likely requires engineering knowledge to know it exists.

There almost always is, actually. If you’re in the cloud and aren’t a tiny startup, that means you’ve had team[s] building your infrastructure, probably led by devs at some point.

It doesn’t take engineering knowledge to browse through CloudWatch metrics and see that your average CPU utilization is in the single digits.

Re: Upgrading Uber's MySQL Fleet

#199

Post by hodgesrm »

ssl-3 wrote: post #92
hodgesrm wrote: post #72

Earlier quoted context omitted.

Not exactly a stylistic difference but there are real differences in the dialects. Here's example from many moons ago: "Even I think that's a bad idea." That was an Indian colleague. It took me weeks to figure out that he was using "even" in place of "also." In a like vein when Australians say "goodeye" they usually aren't talking about your vision.

Perhaps. Or perhaps it was meant to specify that they, themselves, might have been presumed to be an outlier who would think it was a good idea, but who has in fact come to think that is a bad idea. Examples of this kind of counter-presumptive use of the word "even": 1: On animals and the weather: "It was so cold that even polar bears were suffering from frostbite and frozen digits." 2: On politics, where one's gener…

In this case it was indeed "also." I've heard it used that way many times.

Another common phrase in Indian English is "do the needful," which is a delightful formulation. Grammarly has a plausible description of how it arose. [0]

[0] https://www.grammarly.com/blog/idioms/do-the-needful/

Re: Upgrading Uber's MySQL Fleet

#200

Post by paulryanrogers »

tiffanyh wrote: post #64
m4r1k wrote: post #18

Uber's collaboration with Percona is pretty neat. The fact that they've scaled their operations without relying on Oracle's support is a testament to the expertise and vision of their SRE and SWE teams. Respect!

Aren't they using Persona in lieu of Oracle. So it's kind of the same difference, no?

Word on the street is Oracle contracts are expensive and hard to cancel, like a deal with the devil. Not sure if their MySQL support is any different than Oracle DB itself
Post reply on HN