161–170 of 215 posts
File under "things you will never need to do if you use cloud services".
At least in RDS, that will be a one-way upgrade ie: no rollback will be possible. That said, you can upgrade one instance at a time in your cluster for a no-downtime rollout.
I can tell from a mile away that this is written by ChatGPT / Claude, at least partially. "This distinction played a crucial role in our upgrade planning and execution strategy." "Navigating Challenges in the MySQL Upgrade Journey" "Finally, minimizing manual intervention during the upgrade process was crucial."
Earlier quoted context omitted.
It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong Don't know about your LLM feeling
It contains the word "delve", a word that got way more popular in use since the introduction of LLMs. Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive: We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before emba…
Earlier quoted context omitted.
VACUUM and VACUUM FULL (and/or with ANALYZE) can lock tables for a very long time, especially when the table is large. Incantation may also require 2x the space for the table being operated on. In short: it's slow.
Only FULL takes a serious lock (normal vacuum only takes a weak lock preventing things like other vacuums or table alterations iirc). Aside: I wish Postgres forced to make explicit the lock taken. Make me write “TAKE LOCK ACCESS EXCLUSIVE VACUUM FULL my_table”, and fail if the lock I take is too weak. Implicit locks are such a massive footgun that have caused countless incidents across the world, it’s just bad design…
Earlier quoted context omitted.
As somebody who has always used MySQL, but always been told that I should be using Postgres, I'd love to understand what the issues with VACUUM are, and what I should be aware of when potentially switching databases?
MySQL stores table data in a b+ tree where updates modify the data directly in place as transactions are committed, and overwritten data is moved to a secondary undo log to support consistent reads. MySQL indexes store primary keys and queries rely on tree traversal to find the row in the b+ tree, but it can also contain references to rows in the undo log. PostgreSQL tables are known as heaps, which consist of slotte…
Not in the heap, but if you have any index on the table (I know, don’t do that for bulk loads, but many don’t / it isn’t feasible sometimes) then you’re still dealing with a B+tree (probably).
Also, MySQL still gets the nod for pure bulk load speed via MySQLShell’s Parallel Import Utility [0]. You can of course replicate this in Postgres by manually splitting the input file and running multiple \COPY commands, but having a tool do it all in one is lovely.
[0]: https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-uti...
Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra r…
https://www.forbes.com/sites/danielnewman/2023/02/21/uber-go...
File under "things you will never need to do if you use cloud services".
That's not true. The RDS 5.7 instances are EOL so you have to upgrade them at some point. At least in RDS, that will be a one-way upgrade ie: no rollback will be possible. That said, you can upgrade one instance at a time in your cluster for a no-downtime rollout.
Why upgrade to v8.0 (old LTS) and not v8.4 (current LTS)? Especially given that end-of-support is only 18-months from now (April 2026) … when end-of-support of v5.7 is what drive them to upgrade in the first place. https://en.m.wikipedia.org/wiki/MySQL
I suppose they opted for a conservative upgrade policy, as v8.4 probably includes all the functional additions/changes of the previous v8.1+ versions, and moving to it would have been a very big step. MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.
I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly.
As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers
Unless you know something we don't and we're just lucky.
I wonder if an upgrade like this would be less painful if the db layer was containerized? The migration process they described would be less painful with k8s. Especially with 2100+ nodes/VMs
The current bottleneck appears to be etcd, boltdb is just a crappy data store. I would really like to try replacing boltdb with something like sqlite or rocksdb as the data persistence layer in etcd but that is non-trivial.
You also start seeing issues where certain k8s operators do not scale either, for example cilium cannot scale past 5k nodes currently. There are fundamental design issues where the cilium daemonset memory usage scales with the number of pods/endpoints in the cluster. In large clusters the cilium daemonset can be using multiple gigabytes of ram on every node in your cluster. https://docs.cilium.io/en/stable/operations/performance/scal...
Anyways, the TL;DR is that at this scale (16k nodes) it is hard to run k8s.
Earlier quoted context omitted.
You’re not renaming tables when you’re at scale.
Sure you do! It's how online schema changes tend to be done, e.g. https://docs.percona.com/percona-toolkit/pt-online-schema-ch... describes doing an atomic rename as the last step.