[flagged]
Postgres data stored in Parquet on S3: LTAP architecture explained
61–66 of 66 posts
Re: Postgres data stored in Parquet on S3: LTAP architecture explained
#62Earlier quoted context omitted.
Correct, but RustFS is the only drop in replacement (just migrated) Garage and Seaweed are nice (didn't look into Ceph) but you have to re-ingest. RustFS was just plug and play albeit a few minor API differences. Also Apache licensing gives some peace of mind after the musical chair license game before they finally landed on only paid AIStor offering.
Don't look into the news about rustfs from the last few months if you are happy with them. :)
there was the issue in Feb but it was resolved, something else newer happen?
Re: Postgres data stored in Parquet on S3: LTAP architecture explained
#63But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs. You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+…
CDC _never_ goes away, that's for sure. Maybe just for replication to a logical replica, but you'll often need a sync from one system to another -- it just ends up happening.
Re: Postgres data stored in Parquet on S3: LTAP architecture explained
#64Earlier quoted context omitted.
Don't look into the news about rustfs from the last few months if you are happy with them. :)
there was the issue in Feb but it was resolved, something else newer happen?
Not that I know of but a lot of people took tat as a sign of extensive vibe coding and wrote Rustfs off completely. I personally installed seaweedfs, rustfs and garage at different points and plan on checking out rustfs again in the future. If anybody is interested, the most robust of them seems to be seaweed by the way.
Re: Postgres data stored in Parquet on S3: LTAP architecture explained
#65Earlier quoted context omitted.
Hmm, if the caching layer doesn't change(I assume it was optimized for olap style queries), and the new parquet format is better for olap... I'm still not understanding how it performs well for oltp reads. I'll give the article another read... Maybe I missed something. Thank you for the response! Really nice to be able to get info straight from people who work on the product
Recent data plus working set is always in Postgres page format. Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path. So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP. You can be smart in querying both representations for real time analytical queries
I assume the parquet files are way larger than the page format, doesn't this cause a lot of read amplification? OLTP side needs to read a lot more data to fetch a single old row that's stored in parquet format.
Re: Postgres data stored in Parquet on S3: LTAP architecture explained
#66Earlier quoted context omitted.
> CDC which is error prone Just have superior CDC :)
Exactly! Why unifying storage, which opens up a can of trade-offs.
zhou here :)
Moonlink was arguably one of the best ZeroETL(mirroring) solution, and there's a reason we built LTAP instead (not just because it's cool and we can do it)
To me, two big issues with CDC/mirroring are: 1. For ad-hoc querying, CDC is usually not there when you need them, and it is too expensive to maintain for all tables if you don't query them often. 2. For true data pipeline, a simple mirror is not enough, you will end up building all kinds of transformation and it essentially becomes ETL like spark.
One small issue is CDC + merge into columnstore means huge write amplification if you want fresh read on Analytics side, it is absurd especially for lakehouse tables (I have seen 100X for many OLTP workloads).
The only solution I found that we can trust agents creating, managing, running OLTP & OLAP queries is LTAP.