Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres data stored in Parquet on S3: LTAP architecture explained

databricks.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#21

Post by tux3 »

khurs wrote: post #17
tux3 wrote: post #14

Earlier quoted context omitted.

The large enterprise vendors are not prise-sensitive. They're on AWS because you never get fired for picking AWS, and there isn't really any other choice for these vendors regardless of AWS ripping you off. At this point S3 is a standard interface. All sorts of cloud providers and open-source projects provide S3. If you're on AWS, price isn't the reason. You pick AWS because you don't see your company taking a risk w…

"The large enterprise vendors are not price-sensitive." Have you ever spoken to a CTO? They most certainly are. Also many are Microsoft houses so using Azure blob plus one of the reasons for Kubernetes/Openshift adoption was to be cloud neutral

There is a scale between prise-sensitivity and risk-averseness, from my point of reference large companies are much more risk-averse than they are price sensitive. Of course this will vary, CTOs exist in all sort of different environments.

Price is not the reason people chose AWS. Some companies use Azure. The current startup at $WORK uses yet another smaller Cloud. And yet AWS sill has the clear lead in market share. That's because price is far from the only factor, and not even the main factor.

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#22

Post by dsauerbrun »

Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes?

Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files?

What's the special sauce here? Seems like they're just caching the data which, for all intents and purposes, seems like the same solution of storing another copy of the data which is what they say they're avoiding.

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#23

Post by conradludgate »

dsauerbrun wrote: post #22

Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and pu…

Hi, I work on Lakebase (but not on storage), here's how I understand it.

For Lakebase and Neon, our architecture needs the caching layer regardless (what we call Pageservers). Performing reads from S3 directly is too slow so we reconstruct pages and keep them on an nvme server for faster querying. Changing the format on S3 to be Parquet effectively introduces no additional copies over our existing architecture

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#24

Post by _zoltan_ »

khurs wrote: post #12
_zoltan_ wrote: post #10

Earlier quoted context omitted.

your reply makes zero sense. there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog. nobody will run minio on AWS other than hobby projects and small demos. I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when yo…

> nobody will run minio on AWS other than hobby projects and small demos. You realise not every company uses AWS for any/all its needs? There are datacenters around the world owned by individual companies or co-located. And many companies still have servers on prem. Compute and disks are getting more dense & liquid cooled, so less rack space is needed for same power. And Minio and others can handle Petabytes+ https:/…

they exist, sure. And I'm sure it can handle PB+. on prem is an existing market, however, if you reread my comment I talked about running minio on AWS because S3 is too expensive - just doesn't make sense to do.

I've yet to met a Fortune 100 who isn't mostly using either on prem or a large hyperscaler (S3/Azure/GCS).

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#25

Post by _zoltan_ »

tux3 wrote: post #14
_zoltan_ wrote: post #10

Earlier quoted context omitted.

your reply makes zero sense. there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog. nobody will run minio on AWS other than hobby projects and small demos. I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when yo…

The large enterprise vendors are not prise-sensitive. They're on AWS because you never get fired for picking AWS, and there isn't really any other choice for these vendors regardless of AWS ripping you off. At this point S3 is a standard interface. All sorts of cloud providers and open-source projects provide S3. If you're on AWS, price isn't the reason. You pick AWS because you don't see your company taking a risk w…

when people talk about S3 they mostly mean AWS, but I do agree S3 is a standard interface. in non-AWS cases they'd say S3-compatible.

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#28

Post by xtracto »

khurs wrote: post #17
tux3 wrote: post #14

Earlier quoted context omitted.

The large enterprise vendors are not prise-sensitive. They're on AWS because you never get fired for picking AWS, and there isn't really any other choice for these vendors regardless of AWS ripping you off. At this point S3 is a standard interface. All sorts of cloud providers and open-source projects provide S3. If you're on AWS, price isn't the reason. You pick AWS because you don't see your company taking a risk w…

"The large enterprise vendors are not price-sensitive." Have you ever spoken to a CTO? They most certainly are. Also many are Microsoft houses so using Azure blob plus one of the reasons for Kubernetes/Openshift adoption was to be cloud neutral

They definitely havent. Tech side of companies is a Cost Center. And the main question the CEO/CFO makes to the CTO every week is "how can we reduce our AWS bill?" , even before the how was your weekend ? One.

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#29

Post by scritty-dev »

So then would LTAP sit to both the left and the right of the medallion architecture? Meaning would you on the left of Bronze use it as an OLTP and to the right of Gold use it as an OLAP? Currently we've been mainly utilizing it to the right of Gold to develop analytic PERN applications that allow us to reuse the RBAC/ACLs set in Unity Catalog, but from this article it seems like that's only half of its utility?

Re: Postgres data stored in Parquet on S3: LTAP architecture explained

#30

Post by scritty-dev »

khurs wrote: post #19
dockerd wrote: post #15

Earlier quoted context omitted.

Minio is no longer maintained.

oh wow, another rug pull!! So Ceph/SeaweedFS/RustFS/Garage are the alternatives I think

Correct, but RustFS is the only drop in replacement (just migrated) Garage and Seaweed are nice (didn't look into Ceph) but you have to re-ingest. RustFS was just plug and play albeit a few minor API differences.

Also Apache licensing gives some peace of mind after the musical chair license game before they finally landed on only paid AIStor offering.

Post reply on HN