Live data from Hacker News

Is 20M of rows still a valid soft limit of MySQL table in 2023?

yishenggong.com

21–30 of 86 posts

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#21
post #5
post #4

You can safely go to a billion rows. Use indexes. Don't do silly things like count(*) or select *, select the small data sets you need using your indexes. If you need a full table count for some reason, use a primary key that auto increments and select the max, or use information_schema for an estimate. Nobody will sweat the slight inaccuracy on a table that size.

yeah i was also under the 20mil impression initially until building a system that worked with a 500mil row table without any problems (just a beefy RDS instance, nothing crazy distributed or anything). Schema changes became a pain at that point, so i would probably split it if I were to do it again, but reads and writes never gave us much trouble.

check out gh-ost migrations: https://github.com/github/gh-ost

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#22
Hardware has limits as well as software. You would need a completely different architecture to accomodate a 20M+ database. Usually schemeless the database overhead is typically what is responsible how many rows it handle and because your client interacts with the MySQL engine it is at limit and performance of that engine. So yeeeee-no. Time to use a different database.

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#23
post #11

Thanks for repost the blog! Some crazy guys have already told me that they have mysql table >1.4B rows (~180GiB), underestimated mysql's power lol

My current employer has a single InnoDB table that is 14-15 TB.

I bet queries take minutes.

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#25

Hardware has limits as well as software. You would need a completely different architecture to accomodate a 20M+ database. Usually schemeless the database overhead is typically what is responsible how many rows it handle and because your client interacts with the MySQL engine it is at limit and performance of that engine. So yeeeee-no. Time to use a different database.

Postgres is also a relational DB and rows per table are "limited by the number of tuples that can fit onto 4,294,967,295 pages"

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#26
post #9

...yes, performance will drop when your set of active data exceeds available memory and your system needs to reference slower storage. I assume this would apply to anything.

but that threshold is surely not generally 20M rows? I mean wouldn't it be completely dependent on how much memory you have?

it's obviously based on the data size itself, memory available etc etc.

Pretty clearly people ran the exact same test, fill a table with a bunch of empty/meaningless data and see where performance degrades - then write a catchy headline title/post.

In the end, it's more nuanced than that but i think the overall theme is know your tuning once you get to certain data sizes.

Re: Is 20M of rows still a valid soft limit of MySQL table in 2023?

#30
post #4

You can safely go to a billion rows. Use indexes. Don't do silly things like count(*) or select *, select the small data sets you need using your indexes. If you need a full table count for some reason, use a primary key that auto increments and select the max, or use information_schema for an estimate. Nobody will sweat the slight inaccuracy on a table that size.

Agreed. I have a handful of tables with over 2 billion rows and have experienced no issues. One database is 2.5TB in size. I feel like 20 million rows is a relatively small MySQL table.

Nice! We're at 9TB... I agree that 20M is small enough that I'm not concerned about performance at all.
Post reply on HN