We get sub-ms response time on a table with over 250 million rows. We have a covering index for all high-volume queries, running on rds (aurora) with 16gb ram.
Is 20M of rows still a valid soft limit of MySQL table in 2023?
81–86 of 86 posts
Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?
#82We get sub-ms response time on a table with over 250 million rows. We have a covering index for all high-volume queries, running on rds (aurora) with 16gb ram.
Do you need to do aggregations/scans on the table as well or is it mostly direct row accesses?
Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?
#83Earlier quoted context omitted.
You run MongoDB as a backing store. Very web much scale.
I prefer/dev/null for write heavy workloads that need long term storage. There’s plenty of data in modern physics to suggest that there’s no information loss from going into a black hole, so there shouldn’t be any problems. Put “architected web scale data singularity with bulletproof disaster recovery plan” in your CV. You don’t need to mention the recovery plan is to interview for new jobs when someone starts asking…
Haha. Good one.
Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?
#84B+ trees scale vertically very well, until they don't.
Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?
#85We get sub-ms response time on a table with over 250 million rows. We have a covering index for all high-volume queries, running on rds (aurora) with 16gb ram.
Do you need to do aggregations/scans on the table as well or is it mostly direct row accesses?
Yeah, we do a lot of those and they do take a lot longer, and with much wider variance. On the order of 500ms-5s. These are all done async so it’s not a problem but we have begun dual writing to clickhouse to get us out of the aggregations business.
Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?
#86We've got MySQL 8 running on high-end-ish consumer hardware, with 150 tables ranging from thousands to billions of rows. Largest table is a bit over 5 billion rows and 520GiB. Parallel network backup takes about 3 hours.
What does that table do?