(2018)
Added above. Thanks!
Updating a 50 terabyte PostgreSQL database (2018)
11–20 of 81 posts
Re: Updating a 50 terabyte PostgreSQL database (2018)
#12Terms like ”master / slaves” carry racial overtones so many prefer terms like ”primary / replicas” . Similarly, ”allowlist / blocklist” rather than ”whitelist / blacklist” .
" Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents. " https://news.ycombinator.com/newsguidelines.html Please don't take HN threads on generic ideological tangents. This point has been argued to death a thousand times here already, and no interesting discussion is going to be had by changing the subject to it. https://…
Ack. I read the article before the 2018 suffix was appended to the link so was initially under the impression that it was a recent article. Was trying to be helpful given they concluded the article by saying they were hiring, and that some minor language adjustments might help them towards that objective. My apologies, definitely not trying to incite controversy.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#13They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
Maybe their concept of downtime is purely related on the transactions log side.
Assuming they're like my bank: they do maintenance like these every X months, and during that time frame ATMs and E-Banking are offline, but transactions keep on working without interruptions.
Not sure if this is the case though
Re: Updating a 50 terabyte PostgreSQL database (2018)
#14They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
They mentioned this:
> One other detail to note is that we built our software architecture in such a way that we can stop traffic to our PostgreSQL databases, queue the transactions,and run a PostgreSQL update without affecting payments acceptance.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#15Terms like ”master / slaves” carry racial overtones so many prefer terms like ”primary / replicas” . Similarly, ”allowlist / blocklist” rather than ”whitelist / blacklist” .
yeah, nah. I still use master / slave and whitelist / blacklist, I'll continue to do so as long as this Orwellian policing of language continues.
Are you saying that when this policing will end you'll stop using master / slave?
Re: Updating a 50 terabyte PostgreSQL database (2018)
#16They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
And how can you at a bank run a transaction if you don't know if there is money or not in the account ???
Re: Updating a 50 terabyte PostgreSQL database (2018)
#17> As PostgreSQL options were not suitable for the next upgrade, in parallel we considered other possibilities. Our storage devices were able to make instant snapshots and also make them available on remote storage devices over the network, within a much smaller timeframe
ZFS?
Re: Updating a 50 terabyte PostgreSQL database (2018)
#18Terms like ”master / slaves” carry racial overtones so many prefer terms like ”primary / replicas” . Similarly, ”allowlist / blocklist” rather than ”whitelist / blacklist” .
I actually sort of agree with retiring master and slave because they were just not good descriptive names. Primary and secondary are more accurate descriptions. But I am still struggling to see an issue with blacklist and whitelist as they actually don’t seem to have anything to do with race, nor have they ever. They have more to do with darkness and light, or death, etc. Like something being a black box just has no…
Quantum supremacy, command execution, whitespace, race condition, Red-black trees, blackboard, n-gram, Domain Specific Language, Domain, Chained iterators, locks.
Jokes aside, what matters is what you mean by words and how do you feel when you realize that what you're saying genuinely offends somebody. Imagine you don't really think about slavery when you say "sell down the river". Somebody points it out to you what it actually means. You can say "oh, shit" and try to avoid the phrase. Or you can say "it's my inalienable right to keep using this sentence". Or you can say "well, yes in this case it makes sense, this sentence is racist, but if we start to police words, where does it stop?"
Let's see; let's replace that sentence with any of the above. Is "whitespace" really a space for white people or is it because space between words is ink-free?
Is a "race condition" a condition of a race of people or a competition of speed? The words are spelled and pronounced the same but they are not the same word. One comes from romance languages and the other from Norse.
But it's not only etymology. Contrast "master" in "master-slave" where it clearly means the owner/controller of the slave and it stems from shameful practices actually happened in our past by and to our ancestors. Contrast this with "master copy", "to master an art", etc. where the word retains its original meaning of "teacher" (magister).
Be smart, be curious, don't stop at the surface.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#19They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
Naively thinking I would assume they're only queueing the updates, but running the read requests against the slaves in that timeframe.
This would mean that there is probably a possibility for people to "overspend" money they don't have, because a recent transaction wasn't yet applied to their account when doing the next transaction quickly afterwards, but if the timeframe is sufficiently short, that might be a risk they're simply willing to tolerate and mitigate by organizational means.
Alternatively one could also integrate some kind of quick-and-dirty solution specifically to catch cases like that, by tracking "temporary balances" in an intermediate layer and denying payments if they are likely to exceed the actual account balance. Such mechanisms wouldn't have to replicate the actual business logic exactly, but just roughly, in order to provide meaningful protection against exploitation of this temporary update situation.
Re: Updating a 50 terabyte PostgreSQL database (2018)
#20They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
This indeed means downtime on database level. But their system design allows processing transactions with DB down for some time. I guess they are caching account balances and limits in some in-memory DB. This is a common design choice for payment processing systems.
Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet. This is one of the few points why commercial DBMSes are often worth their money.