Live data from Hacker News

Updating a 50 terabyte PostgreSQL database (2018)

medium.com

71–80 of 81 posts

Re: Updating a 50 terabyte PostgreSQL database (2018)

#71
post #8

Earlier quoted context omitted.

This is where I'm at too. Totally on board for retiring the use of master/slave for primary/replica. But like you said, whitelist/blacklist didn't have anything to do with race. I just don't see why it's relevant at all to the conversation when we are talking about terms or names that may be racially insensitive. It's on you to bring up evidence for why a phrase is insensitive. I've yet to see any.

The argument is that the underlying meaning of whitelist being “ok” and blacklist being “not ok” implies some bias based on color. It wasn’t long ago that there were “whites only” and “no blacks allowed” signs throughout the US. For foreign speakers using a more descriptive name like allowlist and denylist is a benefit as well.

These references have everything to do with presence and absence of light, not race.

We are diurnal so we mostly see light as a positive (daylight, when we are awake), and darkness as a negative (night when we are asleep, and have evolved to be more wary of a situation where we can’t see well).

My argument for all this is, instead of actually vilifying the colors themselves which are meaningless, why don’t we stop referring to humans as white and black? Where’s that debate? Because those words are meaningless. Neither are white nor black. They’re brown. We are all different shades of brown.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#72
post #46

Earlier quoted context omitted.

You assume too much. For example, you assume that analytics aren't already run on a separate data warehouse. Ease of accessing older data is an important aspect of database holding transaction (not meaning transactional db). Wouldn't you want to check your transactions on bank page that are older than 30 days?

Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format or data archiving capabilities available in 2021. It's OK! I was in the same boat until I needed to implement such a solution for my use case. What I'm suggesting does not limit their customers from searching any historical data. Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested…

> Your comment clearly illustrates that you have no working knowledge

Please omit swipes like that from your comments here. The rest of your comment would be find without that bit.

https://news.ycombinator.com/newsguidelines.html

Re: Updating a 50 terabyte PostgreSQL database (2018)

#73
post #46

Earlier quoted context omitted.

Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format or data archiving capabilities available in 2021. It's OK! I was in the same boat until I needed to implement such a solution for my use case. What I'm suggesting does not limit their customers from searching any historical data. Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested…

>Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested solution. That must be trolling. 1000x faster than single digit millisecond indexed query retrieving 15 rows? The fact that you keep talking about storage size means that you're talking about analytics not transactional needs. >Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format…

> fuck off with that condescension

Please follow the site guidelines, regardless of whether or not someone else has broken them. Otherwise we just get a downward spiral.

https://news.ycombinator.com/newsguidelines.html

Re: Updating a 50 terabyte PostgreSQL database (2018)

#74
post #73

Earlier quoted context omitted.

>Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested solution. That must be trolling. 1000x faster than single digit millisecond indexed query retrieving 15 rows? The fact that you keep talking about storage size means that you're talking about analytics not transactional needs. >Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format…

> fuck off with that condescension Please follow the site guidelines, regardless of whether or not someone else has broken them. Otherwise we just get a downward spiral. https://news.ycombinator.com/newsguidelines.html

Sorry, I get way too heated when I see this type of condescending argumentation.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#75
post #24

I would probably implement an "archive" system that stores static data that is mostly accessed by "sum" functions - In order to keep the main database relatively small. If the data is immutable/static replication and caching becomes much easier.

I'm not sure what you mean by sum functions, but Postgres natively support foreign tables, and I'm aware of one bank that uses this for older, immutable data.

With "sum" functions, I mean that for money transactions the data will likely be stored in columns and the most common operation will likely be to sum each column. So you could store an "archive" with all the columns/transactions, then store just the sum of each column in the "hot" database. If the data is static/immutable you have a lot of options when it comes to caching and optimizing different queries.

Another strategy would also be to "shard" the database... I guess storing everything in the same database is the most simple solution, but problems will arise when you have to replicate/recover (an arbitrary) 100+ TB of data. Just copying it over a 100Gbit link will take 3 hours.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#76
post #73

Earlier quoted context omitted.

> fuck off with that condescension Please follow the site guidelines, regardless of whether or not someone else has broken them. Otherwise we just get a downward spiral. https://news.ycombinator.com/newsguidelines.html

Sorry, I get way too heated when I see this type of condescending argumentation.

Yes, I experience that too and I bet most people do. One way I often explain this that seems to work well is this:

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Re: Updating a 50 terabyte PostgreSQL database (2018)

#77

> As PostgreSQL options were not suitable for the next upgrade, in parallel we considered other possibilities. Our storage devices were able to make instant snapshots and also make them available on remote storage devices over the network, within a much smaller timeframe ZFS?

It might be NetApp.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#78
post #77

> As PostgreSQL options were not suitable for the next upgrade, in parallel we considered other possibilities. Our storage devices were able to make instant snapshots and also make them available on remote storage devices over the network, within a much smaller timeframe ZFS?

It might be NetApp.

Ah thanks -- I was wondering why they wouldn't mention it if it was ZFS, would have made a good addition to the article. I guess they didn't say because it was something proprietary?

Re: Updating a 50 terabyte PostgreSQL database (2018)

#79
post #77

Earlier quoted context omitted.

It might be NetApp.

Ah thanks -- I was wondering why they wouldn't mention it if it was ZFS, would have made a good addition to the article. I guess they didn't say because it was something proprietary?

No idea, I actually came to the comments for the same reason: looking for details about the storage.

In my opinion it's either zfs or netapp. Zfs can replicate datasets via zfs send, netapp has a snapmirror functionality that does basically the same.

Also, iirc, netapp is contributor to freebsd, so it might be zfs anyway underneath.

I wouldn't be surprised. Last time I had the pleasure to create a snapshot for a volume in a netapp it kinda felt like creating a zfs snapshot, in term of speed and ease.

I thought of netapp since they're running fancy 768gb boxes. If they spend money well on their hardware, they probably spend money well on their storage too.

Re: Updating a 50 terabyte PostgreSQL database (2018)

#80
post #79

Earlier quoted context omitted.

Ah thanks -- I was wondering why they wouldn't mention it if it was ZFS, would have made a good addition to the article. I guess they didn't say because it was something proprietary?

No idea, I actually came to the comments for the same reason: looking for details about the storage. In my opinion it's either zfs or netapp. Zfs can replicate datasets via zfs send, netapp has a snapmirror functionality that does basically the same. Also, iirc, netapp is contributor to freebsd, so it might be zfs anyway underneath. I wouldn't be surprised. Last time I had the pleasure to create a snapshot for a volu…

Yeah zfs send (and a bunch of other features/architecture choices) have been one of the biggest reasons I'm interested in the zfs ecosystem and kind of keep a lookout for it/finding ways to work it into infrastructure that I build. I know that Hardware RAID is what the big boys use, but assuming the performance hit is OK, the feature set of ZFS with a properly tuned database installation like a match made in heaven, was hoping to have them comment...
Post reply on HN