Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

121–130 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#121

File under "things you will never need to do if you use cloud services".

Nah random outages because the RDS instance you were on decided to faceplant itself, or the weird memory to bandwidth scaling AWS has chosen will make you pull your hair out on a high traffic day. It's just different problems.

The company in the article is doing < 200qps per node. Unless they are returning a feature-length video file from every query, they are nowhere near any hardware resource limits.

Re: Upgrading Uber's MySQL Fleet

#122

Wait until they find out they have to upgrade to 8.4 now

And also all the passwords away from mysql_native_password

Which one should do anyways. mysql_native_password is considered broken for a out ten years. (Broken for people who can access the hashed form of the password on the server)

Re: Upgrading Uber's MySQL Fleet

#123
post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong Don't know about your LLM feeling

It contains the word "delve", a word that got way more popular in use since the introduction of LLMs.

Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive:

    We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before embarking on each stage of the version upgrade, we actively involved the on-call teams responsible for each cluster, fostering collaboration and ensuring comprehensive oversight.
The paragraph uses "commencing from" together with "descending to". People would probably write something like "starting with". It shows how the LLM has no spatial understanding: tier 0 is not below or above tier 5, especially as the text has not introduced any such spatial ordering previously. And it gets worse: there is no prior mention of the word "tier" in the blog post. The earlier text speaks of stages, and lists 5 steps (without giving them any name, but the standard term is more like "step" instead of "tier").

There is more signs like "embark", or that specific use of "fostering collaboration" which goes beyond corporate-speak, it also sounds a lot like what an LLM would say. Apparently "safeguard" is also a word LLMs write very often.

Re: Upgrading Uber's MySQL Fleet

#124
post #106

Earlier quoted context omitted.

Didn't realize their entire MySQL data layer runs in AWS. Given that they went with basically a blue-green update strategy, this was, essentially a "witness our cloud spend" kind of post.

They're not. Almost all of their infra was on prem when I worked there 3 years ago.

It's neither. I remember them moving to the cloud but apparently they moved to Google/Oracle (the latter making this article particularly interesting btw). As per the relevant press release : "It’s understood that Uber will close down its own on-premises data centers and move the entirety of its information technology workloads to Oracle and Google Cloud."

Re: Upgrading Uber's MySQL Fleet

#125
post #86

I wonder if an upgrade like this would be less painful if the db layer was containerized? The migration process they described would be less painful with k8s. Especially with 2100+ nodes/VMs

upgrade clients and testing the application logic, changes to the queries themselves as written, the process of detecting the regression and getting MySQL patched by percona, changes to default collation ... all of these things have nothing to do with whether the instances are in containers and whether the containers are managed by k8s or not.

Re: Upgrading Uber's MySQL Fleet

#126
post #123

Earlier quoted context omitted.

It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong Don't know about your LLM feeling

It contains the word "delve", a word that got way more popular in use since the introduction of LLMs. Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive: We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before emba…

Relevant pg thread on twitter: https://x.com/paulg/status/1777030573220933716

Re: Upgrading Uber's MySQL Fleet

#127

Having spent a couple months doing a corporate mandated password rotation on our services - a number of which weren't really designed for password rotation - happy to see the dual password thing mentioned. Being able to load in a new password while the current one is active is where it's at! Trying to coordinate a big bang where everyone flips over at the same time is misery, and I spent a bunch of time updating serv…

I can't answer with an overview on who got such a feature, but "every" system got a different way of doing that: rotating usernames as well. Create a new user with new password.

This isn't 100% equal as ownership (thus permissions with DEFINER) in stored procedures etc. needs some thought, but bad access using outdated username is simpler to trace (as username can be logged etc. contrary to passwords; while MySQL allows for tracing using performance_schema logging incl. user defined connection attributes which may ease finding the "bad" application)

Re: Upgrading Uber's MySQL Fleet

#129
post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

Let's delve into why you think that

It's simple. Human writing is short and to the point (either because they're lazy or want to save the reader's time), yet still manages to capture your attention. AI writing tends to be too elaborate and lacks a sense of "self".

I feel like this article challenges my patience and attention too much, there is really no need to focus on the pros of upgrading here. We reader just want to know how they managed to upgrade at that large scale, challenges they faced and how the solved them. Not to mention any sane tech writers that value their time wouldn't write this much.

Re: Upgrading Uber's MySQL Fleet

#130
post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong Don't know about your LLM feeling

Nah this isn't a big word salad issue. The content is fine. It's just clearly a text written by humans and then rewritten by an LLM, potentially due to the original author(s) not being native speakers. If you feel it's natural English that's fine too ;)
Post reply on HN