Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updating a 50 terabyte PostgreSQL database (2018)

medium.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: Updating a 50 terabyte PostgreSQL database (2018)

#21

Post by mrkeen »

julienfr112 wrote: post #16
fastest963 wrote: post #3

They're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?

And how can you at a bank run a transaction if you don't know if there is money or not in the account ???

1) Risk management. Accept some losses if necessary.

2) Overdraft. Take the money, putting the account below zero, and fine the customer for letting their account go below zero.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#22

Post by zdkl »

ithkuil wrote: post #15
lurker3901 wrote: post #4

Earlier quoted context omitted.

yeah, nah. I still use master / slave and whitelist / blacklist, I'll continue to do so as long as this Orwellian policing of language continues.

Are you saying that when this policing will end you'll stop using master / slave?

> ithkuil

Critiquing semantic vagueness? Username checks out :)

Re: Updating a 50 terabyte PostgreSQL database (2018)

#23

Post by gdgdgdgbgfff »

Vertical scaling like this will have quite a few physical limits, eg max cores (is it 64?). I would think that they may be fast approaching these limits?. For a an scalable enterprise solution would it not be better, and also more cost-efficient, to horizontally scale. Shard the tables over many nodes.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#24

Post by z3t4 »

I would probably implement an "archive" system that stores static data that is mostly accessed by "sum" functions - In order to keep the main database relatively small. If the data is immutable/static replication and caching becomes much easier.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#25

Post by roywashere »

mrkeen wrote: post #21
julienfr112 wrote: post #16

Earlier quoted context omitted.

And how can you at a bank run a transaction if you don't know if there is money or not in the account ???

1) Risk management. Accept some losses if necessary. 2) Overdraft. Take the money, putting the account below zero, and fine the customer for letting their account go below zero.

Just make sure you can cache the last known state of the database and use a separate queue for all non-database-applied transactions. This way you can do database maintenance or suffer database outages without affecting your service.

I heard that around 10 years ago, in The Netherlands a major bank still only wrote transactions to their system in a batch once a day. And all transactions of _today_ showing up in their e-banking applications came from their queue and caching systems on the side of their big database. It's weird but also has some benefits

Re: Updating a 50 terabyte PostgreSQL database (2018)

#26

Post by anonfunction »

Tostino wrote: post #8
atonse wrote: post #7

Earlier quoted context omitted.

I actually sort of agree with retiring master and slave because they were just not good descriptive names. Primary and secondary are more accurate descriptions. But I am still struggling to see an issue with blacklist and whitelist as they actually don’t seem to have anything to do with race, nor have they ever. They have more to do with darkness and light, or death, etc. Like something being a black box just has no…

This is where I'm at too. Totally on board for retiring the use of master/slave for primary/replica. But like you said, whitelist/blacklist didn't have anything to do with race. I just don't see why it's relevant at all to the conversation when we are talking about terms or names that may be racially insensitive. It's on you to bring up evidence for why a phrase is insensitive. I've yet to see any.

The argument is that the underlying meaning of whitelist being “ok” and blacklist being “not ok” implies some bias based on color. It wasn’t long ago that there were “whites only” and “no blacks allowed” signs throughout the US. For foreign speakers using a more descriptive name like allowlist and denylist is a benefit as well.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#28

Post by pupdogg »

Based on their numbers, they should be archiving their historical data in parquet format partitioned by YYYYMMDD onto something like Clickhouse. This way, they can run a lean Postgres instance(s) at all times yet still get benefits of real-time reporting. Based on their use case, they can retain up-to 30 days of data in Postgres and offload the rest onto Clickhouse.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#29

Post by cabraca »

janandonly wrote: post #27

Am I the only one who is shocked to discover that payments (even very small sums) are stored for years on end, maybe indefinite?? Simple deleting a row that is 366 days old is not an option to keep the PostgreSQL DB relatively small?

Thats not shocking. Its probably due to regulatory laws that transactions are saved that long.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#30

Post by wdb »

janandonly wrote: post #27

Am I the only one who is shocked to discover that payments (even very small sums) are stored for years on end, maybe indefinite?? Simple deleting a row that is 366 days old is not an option to keep the PostgreSQL DB relatively small?

It’s a regulatory requirement keep it for 5-7 years e.g for disputes etc
Post reply on HN