Launch HN: Artie (YC S23) – Real time data replication to data warehouses • Hacker News Board

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#31

Post by mbesto » Mon, Jul 24, 2023, 7:08 PM UTC

tang8330 wrote: post #29

slotrans wrote: post #26

(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…

Thanks for your feedback! > Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed eve…

I think you missed the parent's point - your USP is real-time replication. So everything you're proposing makes it not real time. Your USP is now worthless (in that context) and you're competitors are numerous.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#32

Post by tang8330 » Mon, Jul 24, 2023, 7:15 PM UTC

mbesto wrote: post #31

tang8330 wrote: post #29

Earlier quoted context omitted.

Thanks for your feedback! > Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed eve…

I think you missed the parent's point - your USP is real-time replication. So everything you're proposing makes it not real time. Your USP is now worthless (in that context) and you're competitors are numerous.

Hm, perhaps I wasn't being clear, apologies for that.

What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#33

Post by debarshri » Mon, Jul 24, 2023, 9:43 PM UTC

tang8330 wrote: post #29

slotrans wrote: post #26

(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…

Thanks for your feedback! > Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed eve…

I agree with the parent point. I also don't think DWH is the primary usecase for your platform.

I have seen architectures where databases are siloed within departments and data has to be replicated across department physical databases in the same network or different, mostly in banks, insurances and old school industries. In this scenario, a daily batch would run that would replicate and populate the tables and kick start business processes. A platform like this would make sense. Another usecase, i can think of is reverse ETL, but there are many tools custom made for that.

As for fraud analysis, there are many vendor tools that does exactly that, asking people to visualize and implement a full blown usecase is hard.

I might be naive I don't see the USP between artie and Airbyte, hevodata, fivetran, stitch etc. and others from a distance.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#34

Post by RyanHamilton » Mon, Jul 24, 2023, 10:31 PM UTC

Good to see more entrants in the real time analytics space. I've been waiting years for most database to embrace streaming. I'm betting on these real time data platforms needing a faster dashboard than tableau... That's what I'm working on https://www.timestored.com/pulse/. You may find it useful to show off demos. We are partnering with some startups like yourself. Email me if interested. And good luck!

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#35

Post by cgio » Mon, Jul 24, 2023, 10:36 PM UTC

I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed, approved etc. so the flexibility of accommodating them on the pipeline is a benefit from an ease of deployment perspective but could also be alarming for data officers who may feel like they are losing a control lever by not getting the pipeline to block when a schema changes outside normal deployment processes. Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#36

Post by tang8330 » Mon, Jul 24, 2023, 10:49 PM UTC

cgio wrote: post #35

I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed,…

Thanks for the comment!

Your comment regarding DDL is interesting.

Today, this is what happens:

1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value).

2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change.

3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it.

^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column.

We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability.

> Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.

Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking.

On the incremental snapshot and recoveries bit, we primarily leverage Debezium's DDD-3 high watermark strategy [1] for MySQL and MongoDB. Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.

[1] https://github.com/debezium/debezium-design-documents/blob/m... [2] https://www.morling.dev/blog/insatiable-postgres-replication...

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#37

Post by LeonB » Mon, Jul 24, 2023, 11:04 PM UTC

cgio wrote: post #35

I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed,…

I found in integration/warehousing that if a source system “suddenly” has a new column — it’s best if we can automatically just bring it in. We would tend to do this in a way that it can’t break the warehouse, doesn’t affect people downstream of us. We can then choose to make it available to others, or not — but the moment that data is available we start hoovering it in.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#38

Post by [deleted] » Mon, Jul 24, 2023, 11:14 PM UTC

slotrans wrote: post #26

(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…

[deleted]

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#39

Post by cgio » Tue, Jul 25, 2023, 12:19 AM UTC

tang8330 wrote: post #36

cgio wrote: post #35

I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed,…

Thanks for the comment! Your comment regarding DDL is interesting. Today, this is what happens: 1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value). 2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-lin…

The custom snapshotter sounds interesting, potentially a good selling point. On recovery end, in my designs I have also found it useful to have synthetic events so in breakage of CDC I can stitch logs together and not just start from scratch and lose history. I can see you are in the depths of it, more than I’ve been for a while. Wish you the best.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#40

Post by cgio » Tue, Jul 25, 2023, 12:27 AM UTC

LeonB wrote: post #37

cgio wrote: post #35

I’ve been in this space and can appreciate your design decisions. They are meaningful, but a couple of comments on the differentiating factors. DML is tackled in traditional pipelines very similarly, I.e. execution of the log at the target, whether that is materialised or as a view. DDL, it is a nice to have from a technical achievement perspective, but changes to a data schema are not live. They have to be deployed,…

I found in integration/warehousing that if a source system “suddenly” has a new column — it’s best if we can automatically just bring it in. We would tend to do this in a way that it can’t break the warehouse, doesn’t affect people downstream of us. We can then choose to make it available to others, or not — but the moment that data is available we start hoovering it in.

That’s the logical thing in the context of this domain. In a broader data domain there are other considerations, not always logical, so if that ends up configurable it’s a feature that broadens application scope.