Viewing profile — danthelion

I've built an interactive demo for CDC to help explain how it works. The app currently shows the transaction log-based and query-based CDC approaches. Change Data Capture (CDC) is …

At Estuary, we’re creating a real-time data streaming platform that doesn’t rely on Kafka and uses JSON as the primary data format stored in an object storage. Many people are inte…

Imagine a system monitoring payment transactions. Each transaction stream (e.g., purchase events) could be joined with customer account data (e.g., past purchasing patterns or blac…

Materialize is powered by timely dataflow and differential dataflow, Estuary Flow uses a streaming mapreduce architecture: https://estuary.dev/why-mapreduce-is-making-a-comeback/

Gazette is at the core of Estuary Flow ( https://estuary.dev ), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays wel…

With the 2-pass strategy, we can write arbitrary row group sizes while using a fixed amount of memory, with probably 100-200 MiB of overhead for the parquet file processing, depend…