I didn't understand how they avoid downtime during the upgrade - they start with "Stop traffic to the database cluster" and then perform a bunch of steps that each take 5-10 minutes or longer before starting traffic again - so do they just accept the downtime and schedule it in advance, or did I miss a detail?
Updating a 50 terabyte PostgreSQL database (2018)
11–20 of 67 posts
Re: Updating a 50 terabyte PostgreSQL database (2018)
#12I didn't understand how they avoid downtime during the upgrade - they start with "Stop traffic to the database cluster" and then perform a bunch of steps that each take 5-10 minutes or longer before starting traffic again - so do they just accept the downtime and schedule it in advance, or did I miss a detail?
> One other detail to note is that we built our software architecture in such a way that we can stop traffic to our PostgreSQL databases, queue the transactions, and run a PostgreSQL update without affecting payments acceptance.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#1350TB is not so big these days. I read that in 2008 (!) Yahoo had a 2+ PB PG database. What is the largest you know of, 14 years later?
Re: Updating a 50 terabyte PostgreSQL database (2018)
#1450TB is not so big these days. I read that in 2008 (!) Yahoo had a 2+ PB PG database. What is the largest you know of, 14 years later?
50TB is big. Bigger is possible I'm sure, but I'd guess 99.something% of all PG databases are less than 50TB.
If someone here commented they had a 2PB database, I guarantee someone else here would be like "pfft, that's not big"...
Re: Updating a 50 terabyte PostgreSQL database (2018)
#15I didn't understand how they avoid downtime during the upgrade - they start with "Stop traffic to the database cluster" and then perform a bunch of steps that each take 5-10 minutes or longer before starting traffic again - so do they just accept the downtime and schedule it in advance, or did I miss a detail?
> One other detail to note is that we built our software architecture in such a way that we can stop traffic to our PostgreSQL databases, queue the transactions, and run a PostgreSQL update without affecting payments acceptance.
That’s downtime by any reasonable measure depending on the exact specifics. Is writing to a journal while the DB is down “downtime”?
Site wise no but DB-wise yes.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#1650TB is not so big these days. I read that in 2008 (!) Yahoo had a 2+ PB PG database. What is the largest you know of, 14 years later?
50TB is big. Bigger is possible I'm sure, but I'd guess 99.something% of all PG databases are less than 50TB. If someone here commented they had a 2PB database, I guarantee someone else here would be like "pfft, that's not big"...
it's more than big enough to cause big problems / risk days of downtime to change, yea. 50GB is not big. 50TB is at least touching big - you can do it on one physical machine if needed, but it's the sort of scale that benefits from bigger-system architecture. 50PB would be world-class big, hitting exciting new problems every time they do something.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#17I didn't understand how they avoid downtime during the upgrade - they start with "Stop traffic to the database cluster" and then perform a bunch of steps that each take 5-10 minutes or longer before starting traffic again - so do they just accept the downtime and schedule it in advance, or did I miss a detail?
Yeah, they architect their application to accept DB downtime – but I'm sure their services are still degraded to some degree or another during this, and they aren't clear how much total DB downtime they need for this (and how that time scales across various axes).
Overall my takeaway is basically "if you want to upgrade a large Postgres db, you'll need like an hour of planned downtime and a lot of careful work" which… doesn't make me excited to upgrade big postgres db's in production.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#18Thanks for sharing this. Really interesting. But a basic question, why not upgrading to PG 13 instead? I am curious about the reasons for staying on an older version of PostgreSQL.
Interesting. Does PG13 let then do this without downtime?
Re: Updating a 50 terabyte PostgreSQL database (2018)
#19One of the things I always wonder with giant relational database is. How much of the "typical relational stuff" are they actually using?
Do they have constraints on rows? Are they using views or do they just denormalize and duplicate? Do they use joins at all? Are they even doing more than 1 thing in a transaction?
Re: Updating a 50 terabyte PostgreSQL database (2018)
#20[deleted]