31–40 of 81 posts
I would probably implement an "archive" system that stores static data that is mostly accessed by "sum" functions - In order to keep the main database relatively small. If the data is immutable/static replication and caching becomes much easier.
Based on their numbers, they should be archiving their historical data in parquet format partitioned by YYYYMMDD onto something like Clickhouse. This way, they can run a lean Postgres instance(s) at all times yet still get benefits of real-time reporting. Based on their use case, they can retain up-to 30 days of data in Postgres and offload the rest onto Clickhouse.
Ease of accessing older data is an important aspect of database holding transaction (not meaning transactional db). Wouldn't you want to check your transactions on bank page that are older than 30 days?
Am I the only one who is shocked to discover that payments (even very small sums) are stored for years on end, maybe indefinite?? Simple deleting a row that is 366 days old is not an option to keep the PostgreSQL DB relatively small?
It’s a regulatory requirement keep it for 5-7 years e.g for disputes etc
Based on their numbers, they should be archiving their historical data in parquet format partitioned by YYYYMMDD onto something like Clickhouse. This way, they can run a lean Postgres instance(s) at all times yet still get benefits of real-time reporting. Based on their use case, they can retain up-to 30 days of data in Postgres and offload the rest onto Clickhouse.
You assume too much. For example, you assume that analytics aren't already run on a separate data warehouse. Ease of accessing older data is an important aspect of database holding transaction (not meaning transactional db). Wouldn't you want to check your transactions on bank page that are older than 30 days?
Earlier quoted context omitted.
You assume too much. For example, you assume that analytics aren't already run on a separate data warehouse. Ease of accessing older data is an important aspect of database holding transaction (not meaning transactional db). Wouldn't you want to check your transactions on bank page that are older than 30 days?
All modern columnnar/analytical databases have no problem in pulling back older data; in fact, that's the point of running large scale analytics.
Earlier quoted context omitted.
" Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents. " https://news.ycombinator.com/newsguidelines.html Please don't take HN threads on generic ideological tangents. This point has been argued to death a thousand times here already, and no interesting discussion is going to be had by changing the subject to it. https://…
Ack. I read the article before the 2018 suffix was appended to the link so was initially under the impression that it was a recent article. Was trying to be helpful given they concluded the article by saying they were hiring, and that some minor language adjustments might help them towards that objective. My apologies, definitely not trying to incite controversy.
Vertical scaling like this will have quite a few physical limits, eg max cores (is it 64?). I would think that they may be fast approaching these limits?. For a an scalable enterprise solution would it not be better, and also more cost-efficient, to horizontally scale. Shard the tables over many nodes.
For the vast majority of businesses, vertical scaling is quite feasible.
Earlier quoted context omitted.
It’s a regulatory requirement keep it for 5-7 years e.g for disputes etc
Just to add to this: GDPR does not apply here.
Vertical scaling like this will have quite a few physical limits, eg max cores (is it 64?). I would think that they may be fast approaching these limits?. For a an scalable enterprise solution would it not be better, and also more cost-efficient, to horizontally scale. Shard the tables over many nodes.
You can buy boxes with 30x more RAM without going super boutique. You can buy quad-socket server motherboards, so you can have 64x4 cores. For the vast majority of businesses, vertical scaling is quite feasible.