Postgres data stored in Parquet on S3: LTAP architecture explained

Post by _zoltan_ » Sat, Jul 04, 2026, 9:41 AM UTC

khurs wrote: post #9

andrenotgiant wrote: post #3

Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?

Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date. SELECT count * FROM my_table AS OF "2025-01-01" https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://iceberg.apache.org/docs/latest/spark-queries/#spark-...

Post by [deleted] » Sat, Jul 04, 2026, 9:57 AM UTC

hasyimibhar wrote: post #8

andrenotgiant wrote: post #3

Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?

It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream. For example, if you know your user can change emails, and there might be events from an…

Post by dockerd » Sat, Jul 04, 2026, 10:45 AM UTC

khurs wrote: post #5

PunchyHamster wrote: post #4

I don't wanna see that S3 bandwidth bill after running some big query

There are self hosted object stores which use the same protocol as S3. One example: https://github.com/minio/minio Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression). And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be…

Post by khurs » Sat, Jul 04, 2026, 11:01 AM UTC

tux3 wrote: post #14

_zoltan_ wrote: post #10

Earlier quoted context omitted.

your reply makes zero sense. there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog. nobody will run minio on AWS other than hobby projects and small demos. I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when yo…

The large enterprise vendors are not prise-sensitive. They're on AWS because you never get fired for picking AWS, and there isn't really any other choice for these vendors regardless of AWS ripping you off. At this point S3 is a standard interface. All sorts of cloud providers and open-source projects provide S3. If you're on AWS, price isn't the reason. You pick AWS because you don't see your company taking a risk w…

Post by khurs » Sat, Jul 04, 2026, 11:10 AM UTC

dockerd wrote: post #15

khurs wrote: post #5

Earlier quoted context omitted.

There are self hosted object stores which use the same protocol as S3. One example: https://github.com/minio/minio Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression). And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be…

Minio is no longer maintained.