41–50 of 66 posts
Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and pu…
Earlier quoted context omitted.
> CDC which is error prone Just have superior CDC :)
Exactly! Why unifying storage, which opens up a can of trade-offs.
This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.
Earlier quoted context omitted.
Exactly! Why unifying storage, which opens up a can of trade-offs.
If you product is CDC based (peerdb) you don’t want storage to support this :) This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.
Separately, I understand taking care of it at storage level, but still don’t get “unifying storage” or “zero copy”.
Anyways, I’ll stop now. Good to see all the innovation happening on converging OLTP/OLAP front. Each with a different approach and perspective. :)
Earlier quoted context omitted.
Exactly! Why unifying storage, which opens up a can of trade-offs.
Did you even read the blog? Or are you just throwing shade because you are working on a competitive product based on CDC?
Earlier quoted context omitted.
It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream. For example, if you know your user can change emails, and there might be events from an…
I think you have a point, and SCD type 2 feels like a workaround, but there is also something to be said for the ability to query every row as it was at any given version. I’m not saying that SCD type 2 is the best solution given there might be a more domain-specific way to do it, but I see it a lot like file-based version control. It’s convenient to be able to examine all files as they existed at any point in time,…
Nope, even if I have the ability to see the exact changes of each row, I would still add timestamps everywhere, because timestamp of row change does not equal event timestamp. For example, if I have an order table with status column, and I see a CDC event where status changed from in_progress to completed, I cannot simply assume that the CDC timestamp is the timestamp when order was completed. It's possible that the source database received the event late a few minutes late due to delay upstream, or it's backfilling some missed orders a few days ago. Having a completed_at timestamp (and a bunch of other timestamps for each order lifecycle) would eliminate any ambiguities, and your data analyst will thank you for it.
It's the same thing with row history. You cannot simply assume that your row changes are aligned with the logical history of your entity.
Earlier quoted context omitted.
If you product is CDC based (peerdb) you don’t want storage to support this :) This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.
;) sounding good on paper vs how it works in practice (supporting demanding real-time OLTP/OLAP workloads) are completely different ball games. Separately, I understand taking care of it at storage level, but still don’t get “unifying storage” or “zero copy”. Anyways, I’ll stop now. Good to see all the innovation happening on converging OLTP/OLAP front. Each with a different approach and perspective. :)
Super cool stuff. Being able to combine your analytical platform and transactional database into one storage layer without having to set up ETL pipelines in between is really a game changer. Especially since it's just postgres, instead of some proprietary database.