War stories of unusually sized databases or performance characteristics?
Ask HN: Largest Postgres DBs?
1–10 of 18 posts
Re: Ask HN: Largest Postgres DBs?
#2A family of posts in this genre for every db would be quite a useful resource.
Re: Ask HN: Largest Postgres DBs?
#3I'm managing a few databases for living, from pg dbs of few hundreds of GBs on-prem, up to few TBs on rds, and almost 1PB cluster of 20 warehouses on Snowflake. Which of those count? (doing data architect/eng for living)
Re: Ask HN: Largest Postgres DBs?
#4I have more experience running large MySQL databases. Both at SurveyMonkey and Zapier our primary databases were MySQL and were massive as you can imagine from massively scaled consumer products. I won't list the data here since this is about postgres but wanted to provide the context.
The largest postgres I've personally administered is on RDS:
* postgres 13.4
* db.m6g.2xlarge (8vCPU, 32 GB RAM)
* 1.3tb storage
* Largest tables: 317GB, 144GB, and 142GB
I have had random bad query plans when using CTEs that get cached and take down the entire DB for a bit but usually been able to fix it with `vacuum analyze` and the customizing the query to convince postgres to choose a better plan.
Overall I think postgres can do a lot of work with very little.
Re: Ask HN: Largest Postgres DBs?
#5I have more experience running large MySQL databases. Both at SurveyMonkey and Zapier our primary databases were MySQL and were massive as you can imagine from massively scaled consumer products. I won't list the data here since this is about postgres but wanted to provide the context. The largest postgres I've personally administered is on RDS: * postgres 13.4 * db.m6g.2xlarge (8vCPU, 32 GB RAM) * 1.3tb storage * La…
For me, your numbers underline the "premature optimization is the root of all evil" saying. Maybe misunderstanding the performance implications of different orders of mangitudes is similar to the lack of understanding in statistics. I often encounter colleagues or customers who worry about some data structure or table when there are 10s of thousands of entries which is actually not enough to spend that much time worrying about.
Re: Ask HN: Largest Postgres DBs?
#6I have a +1tb Postgres database at home that has market data in it. Moving it over to another Postgres db that has timeseries extensions to compact and increase performance. Also thinking about just partitioning everything up into parquet files. Query performance isn’t the best and am currently trying to improve it.
Re: Ask HN: Largest Postgres DBs?
#7I really would like to know details on how people backup, dump, migrate and upgrade DBs of this size.
Re: Ask HN: Largest Postgres DBs?
#8I have more experience running large MySQL databases. Both at SurveyMonkey and Zapier our primary databases were MySQL and were massive as you can imagine from massively scaled consumer products. I won't list the data here since this is about postgres but wanted to provide the context. The largest postgres I've personally administered is on RDS: * postgres 13.4 * db.m6g.2xlarge (8vCPU, 32 GB RAM) * 1.3tb storage * La…
Out of curiosity, were those CTE issues on the current PG version (13.4)? I know they changed some materialization stuff with them in 12 or so but was wondering if that issue was from back then or if you're running into some stuff that currently still happens.
Re: Ask HN: Largest Postgres DBs?
#9I really would like to know details on how people backup, dump, migrate and upgrade DBs of this size.
Incrementally usually :)
Re: Ask HN: Largest Postgres DBs?
#10I have a +1tb Postgres database at home that has market data in it. Moving it over to another Postgres db that has timeseries extensions to compact and increase performance. Also thinking about just partitioning everything up into parquet files. Query performance isn’t the best and am currently trying to improve it.
I know we're talking about postgres but have you looked at clickhouse? it seems pretty promising with big data (and also timeseries data with its log engines)