Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Uber's MySQL Fleet

uber.com

Post reply on HN
Print view View original

201–210 of 215 posts

Re: Upgrading Uber's MySQL Fleet

#201

Post by paulryanrogers »

takeda wrote: post #99
donatj wrote: post #63

Interestingly we just went through basically the same upgrade just a couple days ago for similar reasons. We run Amazon Aurora MySQL and Amazon is finally forcing us to upgrade to 8.0. We ended up spinning up a secondary fleet and bin log replicating from our 5.7 master to the to-be 8.0 master until everything made the switch over. I was frankly surprised it worked, but it did. It went really smoothly.

AFAIK the 8.0 release is one where Oracle breaks compatibility. So anyone considering MariaDB needs to switch before going to 8.0, otherwise switching will be much more painful.

IIRC it wasn't a big break from 5.7 unless you used some odd grouping or relied on its old+lossy defaults.

Though the single threaded performance fell off a cliff, tanking my CI performance.

Maria DB evolved very differently. I'm not sure how they stack up performance wise.

Re: Upgrading Uber's MySQL Fleet

#202

Post by paulryanrogers »

hu3 wrote: post #168
pizza234 wrote: post #14

Earlier quoted context omitted.

I suppose they opted for a conservative upgrade policy, as v8.4 probably includes all the functional additions/changes of the previous v8.1+ versions, and moving to it would have been a very big step. MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.

> MySQL is very unstable software I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly. As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers Unless you know something we don't and we're just lucky.

Depends on your use case. I had a huge table get corrupted after a drop-swap rename during some DDL. 8.0.14 was the version, IIRC.

For simple use cases or with tried and true patch sets I'm sure it can be a work horse.

Re: Upgrading Uber's MySQL Fleet

#204

Post by vbezhenar »

InsideOutSanta wrote: post #31
hu3 wrote: post #13

Earlier quoted context omitted.

Yeah some numbers caught my attention like ~94% reduction in overall database lock time. And to think they never have to worry about VACUUM. Ahh the peace.

As somebody who has always used MySQL, but always been told that I should be using Postgres, I'd love to understand what the issues with VACUUM are, and what I should be aware of when potentially switching databases?

When postgres deletes a row, it marks a disk space unused. If you delete a lot of rows, there will be plenty of unused space, but that space will not be released to OS. To release that space to OS, you need to run vacuum full or perform backup/truncate/restore.

In normal circumstances it's not needed. The unused space is reused for further inserts and if your database maintains steady rate of inserts/deletes, this won't be an issue.

However I experienced situation when database starts to grow uncontrollably and it was happening for weeks (when actual workload didn't change). I don't know what causes that behaviour. The solution was to run `vacuum full`.

Re: Upgrading Uber's MySQL Fleet

#205

Post by lenerdenator »

greenie_beans wrote: post #55
lenerdenator wrote: post #42

Earlier quoted context omitted.

Their focus is prioritized according to what returns maximum value to their shareholders.

beep boop i'm a capitalist robot pretty sure safe travels is critical to maximum value to their shareholders (aka stfu or tell me how this blog post has anything to do with maximize shareholder value https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/... ... shareholder value is a dumb ass thing to prioritize over human life)

> pretty sure safe travels is critical to maximum value to their shareholders

Well, it is... sort of.

Obviously you can't have Uber be a guaranteed way to be robbed by a highwayman, but when you've cleared out most taxis in a given city, you can start to dictate the terms by which customers accept your service.

And if that means including language in your ToS that shove your customers into a binding arbitration agreement [0] that effectively shield you from the risks of hiring incompetent or malicious drivers, well... that's what that means.

[0]https://www.npr.org/2024/10/02/nx-s1-5136615/uber-car-crash-...

Re: Upgrading Uber's MySQL Fleet

#206

Post by lenerdenator »

Kennnan wrote: post #66
greenie_beans wrote: post #55

Earlier quoted context omitted.

beep boop i'm a capitalist robot pretty sure safe travels is critical to maximum value to their shareholders (aka stfu or tell me how this blog post has anything to do with maximize shareholder value https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/... ... shareholder value is a dumb ass thing to prioritize over human life)

Honest question, how do you (or amyone) propose to vet drivers? They require drivers license and car insurance registration, anything like a CDL would make being a driver prohibitively expensive. Their rating system already works as a good signal the few times Ive used uber.

> They require drivers license and car insurance registration, anything like a CDL would make being a driver prohibitively expensive.

Taxi companies made it work.

They just accepted less return on their investment than the tech bros behind Uber did.

Re: Upgrading Uber's MySQL Fleet

#207

Post by drastic_fred »

Wow! Less than 200qps per node, quite a redundancy. They also mentioned its read heavy. With aws dynamo db it would be at least 10 times cheaper for their given workload. Although, i do not know how much data they store and serve or how they are additionally (analytics?, etc) using this fleet for.

Re: Upgrading Uber's MySQL Fleet

#208

Post by donatj »

paulryanrogers wrote: post #201
takeda wrote: post #99

Earlier quoted context omitted.

AFAIK the 8.0 release is one where Oracle breaks compatibility. So anyone considering MariaDB needs to switch before going to 8.0, otherwise switching will be much more painful.

IIRC it wasn't a big break from 5.7 unless you used some odd grouping or relied on its old+lossy defaults. Though the single threaded performance fell off a cliff, tanking my CI performance. Maria DB evolved very differently. I'm not sure how they stack up performance wise.

You can re-enable the old group by behavior, which we did.

Re: Upgrading Uber's MySQL Fleet

#209

Post by greenie_beans »

lenerdenator wrote: post #205
greenie_beans wrote: post #55

Earlier quoted context omitted.

beep boop i'm a capitalist robot pretty sure safe travels is critical to maximum value to their shareholders (aka stfu or tell me how this blog post has anything to do with maximize shareholder value https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/... ... shareholder value is a dumb ass thing to prioritize over human life)

> pretty sure safe travels is critical to maximum value to their shareholders Well, it is... sort of. Obviously you can't have Uber be a guaranteed way to be robbed by a highwayman, but when you've cleared out most taxis in a given city, you can start to dictate the terms by which customers accept your service. And if that means including language in your ToS that shove your customers into a binding arbitration agree…

you can rationalize your way around the issue all day but it still don't make it right.

Re: Upgrading Uber's MySQL Fleet

#210

Post by lawrjone »

remon wrote: post #103

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

Yeah I found this really off putting: it’s not possible for you to have several goals that are all ‘paramount’, and the word ‘seamless’ adds nothing in every place it appears!

I wish it didn’t turn me off the content as much as it does but it’s very jarring.

Post reply on HN