Updating a 50 terabyte PostgreSQL database (2018)

Post by jdmichal » Mon, Mar 22, 2021, 3:31 PM UTC

Slartie wrote: post #19

fastest963 wrote: post #3

They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?

Naively thinking I would assume they're only queueing the updates, but running the read requests against the slaves in that timeframe. This would mean that there is probably a possibility for people to "overspend" money they don't have, because a recent transaction wasn't yet applied to their account when doing the next transaction quickly afterwards, but if the timeframe is sufficiently short, that might be a risk t…

Post by Tostino » Mon, Mar 22, 2021, 3:32 PM UTC

anonfunction wrote: post #26

Tostino wrote: post #8

Earlier quoted context omitted.

This is where I'm at too. Totally on board for retiring the use of master/slave for primary/replica. But like you said, whitelist/blacklist didn't have anything to do with race. I just don't see why it's relevant at all to the conversation when we are talking about terms or names that may be racially insensitive. It's on you to bring up evidence for why a phrase is insensitive. I've yet to see any.

The argument is that the underlying meaning of whitelist being “ok” and blacklist being “not ok” implies some bias based on color. It wasn’t long ago that there were “whites only” and “no blacks allowed” signs throughout the US. For foreign speakers using a more descriptive name like allowlist and denylist is a benefit as well.

Post by prirun » Mon, Mar 22, 2021, 3:57 PM UTC

roywashere wrote: post #25

mrkeen wrote: post #21

Earlier quoted context omitted.

1) Risk management. Accept some losses if necessary. 2) Overdraft. Take the money, putting the account below zero, and fine the customer for letting their account go below zero.

Just make sure you can cache the last known state of the database and use a separate queue for all non-database-applied transactions. This way you can do database maintenance or suffer database outages without affecting your service. I heard that around 10 years ago, in The Netherlands a major bank still only wrote transactions to their system in a batch once a day. And all transactions of _today_ showing up in their…

Post by peterthehacker » Mon, Mar 22, 2021, 4:30 PM UTC

mathnode wrote: post #65

ComodoHacker wrote: post #20

Earlier quoted context omitted.

This indeed means downtime on database level. But their system design allows processing transactions with DB down for some time. I guess they are caching account balances and limits in some in-memory DB. This is a common design choice for payment processing systems. Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet. This is one of the few points why commercial DBMSes are often worth their mone…

MariaDB has offered online schema changes in InnoDB for several years now with continous improvement. Their (currently commercial only) engine, Xpand (previously known as Clustrix), offers distributed SQL as well as online schema changes. No need for an unncessary database migration if you are already on LAMP and the like to get these fancy features. However handling any kind of change should be handled with care, es…

Post by atonse » Mon, Mar 22, 2021, 5:01 PM UTC

ithkuil wrote: post #18

atonse wrote: post #7

Earlier quoted context omitted.

I actually sort of agree with retiring master and slave because they were just not good descriptive names. Primary and secondary are more accurate descriptions. But I am still struggling to see an issue with blacklist and whitelist as they actually don’t seem to have anything to do with race, nor have they ever. They have more to do with darkness and light, or death, etc. Like something being a black box just has no…

Quantum supremacy, command execution, whitespace, race condition, Red-black trees, blackboard, n-gram, Domain Specific Language, Domain, Chained iterators, locks. Jokes aside, what matters is what you mean by words and how do you feel when you realize that what you're saying genuinely offends somebody. Imagine you don't really think about slavery when you say "sell down the river". Somebody points it out to you what…