How are you performing initial sync?
Launch HN: Artie (YC S23) – Real time data replication to data warehouses
51–58 of 58 posts
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#52In 2016, I wrote a pretty detailed answer for "Spark vs. Redshift" question. This was in the very early days of what today I guess is called "the modern data stack" The core of the answer was that cloud warehouses are not suitable for real-time use cases, because the batch processing and transformations take too long. If you want real-time, you need to pay up - hence Databricks / Spark. I did call out the fraud use c…
Thank you so much for the support!
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#53How are you performing initial sync?
For Postgres, we have our own custom snapshotter that is capable of doing parallel snapshots against your read replica and not incur WAL growth. More details here: https://news.ycombinator.com/item?id=36855338
For MySQL and MongoDB, we rely on Debezium to perform the initial snapshots.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#54Can this also be used with a data lake or the data bricks I used to build a data mart?
Definitely. For staging temporary tables to merge for our Redshfit destination, we're uploading it to S3.
We will be creating a S3 destination with TSV, Avro, Parquet format support verily soon.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#55Hi, great project! I was wondering if you are planning on extending support for dynamodb
Hi! Yes, our next 2 connectors are going to be S3 and DynamoDB.
GitHub Issue: https://github.com/artie-labs/transfer/issues/157
If you'd like to be another design partner for us on this, do reach out. I'm at robin@artie.so.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#56This will be game changer. I love this so much - it will increase the speed and reliability of my data! Sent over an email to book a meeting.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#57I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed,…
Thanks for the comment! Your comment regarding DDL is interesting. Today, this is what happens: 1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value). 2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-lin…
> Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.
Does Debezium's DDD-3 watermark (DBLog) implementation for Postgres not process the WAL quickly enough? We don't use it ourselves either, but architecturally it appears it would reasonably bound how long the WAL can remain un-read?
Agreed that many production DBs people care about have pretty severe limitations here! Managed Supabase is another good example.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#58Earlier quoted context omitted.
Thanks for the comment! Your comment regarding DDL is interesting. Today, this is what happens: 1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value). 2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-lin…
> Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL. Does Debezium's DDD-3 watermark (DBLog) implementation for Postgres not process the WAL quickly enough? We don't use it ourselves eit…
On a single unbounded (CPU + mem) Debezium running on a VM extracting Postgres, I was able to clock in about 7-10m/hr. You could increase the # of tasks, but then it'll hinder your DB perf. Also, this is on your primary DB.
We found it far more efficient and less risky to do CDC streaming and snapshotting w/o read lock in parallel to two different topics. Once snapshot is done and drained, we then move to drain the CDC topic.