Launch HN: Artie (YC S23) – Real time data replication to data warehouses • Hacker News Board

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#11

Post by tang8330 » Mon, Jul 24, 2023, 3:11 PM UTC

bpicolo wrote: post #9

> the partition key is the primary key(s) to ensure no out of order writes. Can you get out of order writes if the pk for a row changes?

Great question! Say this happened:

* CREATE TABLE user_test (id int primary key, name text);

* INSERT INTO user_test (id, name) VALUES (1, 'foo');

* UPDATE user_test set id = 2 where id = 1;

When the UPDATE call is invoked, there will be 2 events emitted to Kafka. 1/ A DELETE event for id = 1

2/ A CREATE event for id = 2

Here's the detailed events that get emitted: https://gist.github.com/Tang8330/7a9450f95fbae486a4393abdd49...

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#12

Post by bpicolo » Mon, Jul 24, 2023, 3:22 PM UTC

tang8330 wrote: post #11

bpicolo wrote: post #9

> the partition key is the primary key(s) to ensure no out of order writes. Can you get out of order writes if the pk for a row changes?

Great question! Say this happened: * CREATE TABLE user_test (id int primary key, name text); * INSERT INTO user_test (id, name) VALUES (1, 'foo'); * UPDATE user_test set id = 2 where id = 1; When the UPDATE call is invoked, there will be 2 events emitted to Kafka. 1/ A DELETE event for id = 1 2/ A CREATE event for id = 2 Here's the detailed events that get emitted: https://gist.github.com/Tang8330/7a9450f95fbae486a43…

Interesting! I'm both surprised and also not surprised it's modeled that way.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#13

Post by halfcat » Mon, Jul 24, 2023, 3:24 PM UTC

Is the value here primarily in low latency updates? i.e. the fraud detection scenario

My main pain points with ETLs are with systems that don’t provide good CDC metrics to begin with, which are not Postgres/etc, which are usually a bolt-on ODBC driver or rate-limited APIs.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#14

Post by tang8330 » Mon, Jul 24, 2023, 3:26 PM UTC

halfcat wrote: post #13

Is the value here primarily in low latency updates? i.e. the fraud detection scenario My main pain points with ETLs are with systems that don’t provide good CDC metrics to begin with, which are not Postgres/etc, which are usually a bolt-on ODBC driver or rate-limited APIs.

Main value props are low latency OLTP data in your DWH and lower DWH ingestion costs.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#15

Post by tang8330 » Mon, Jul 24, 2023, 3:28 PM UTC

bpicolo wrote: post #12

tang8330 wrote: post #11

Earlier quoted context omitted.

Great question! Say this happened: * CREATE TABLE user_test (id int primary key, name text); * INSERT INTO user_test (id, name) VALUES (1, 'foo'); * UPDATE user_test set id = 2 where id = 1; When the UPDATE call is invoked, there will be 2 events emitted to Kafka. 1/ A DELETE event for id = 1 2/ A CREATE event for id = 2 Here's the detailed events that get emitted: https://gist.github.com/Tang8330/7a9450f95fbae486a43…

Interesting! I'm both surprised and also not surprised it's modeled that way.

Eventual consistency FTW!

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#16

Post by btown » Mon, Jul 24, 2023, 3:46 PM UTC

tang8330 wrote: post #6

joelhaasnoot wrote: post #5

Do you support PostGIS? This seems to be an issue with a lot of solutions I've seen...

Not at the moment, but no reason(s) why we shouldn't. Debezium already supports this [1] and it'd be a minor refactor on our end to support. If that's something you'd want, LMK! [1] https://debezium.io/documentation/reference/stable/connector...

Seconding this! We're in the short-term rental space, and while properties to don't change locations frequently (though mistakes get corrected, and also https://arkup.com/ exists...), the polygon definitions and hierarchical relationships of regions-of-interest for search and discovery are often being updated, and being able to have real-time geometry intersections rather than rerunning scripts periodically to rebuild our analyses would be incredibly powerful. And certainly for use cases like trucking fleet management etc., real-time analytics on location data across the entire gamut is vital.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#17

Post by tang8330 » Mon, Jul 24, 2023, 3:53 PM UTC

btown wrote: post #16

tang8330 wrote: post #6

Earlier quoted context omitted.

Not at the moment, but no reason(s) why we shouldn't. Debezium already supports this [1] and it'd be a minor refactor on our end to support. If that's something you'd want, LMK! [1] https://debezium.io/documentation/reference/stable/connector...

Seconding this! We're in the short-term rental space, and while properties to don't change locations frequently (though mistakes get corrected, and also https://arkup.com/ exists...), the polygon definitions and hierarchical relationships of regions-of-interest for search and discovery are often being updated, and being able to have real-time geometry intersections rather than rerunning scripts periodically to rebuil…

That's fascinating! Thanks for providing more color, support for geometric shapes coming! https://github.com/artie-labs/transfer/issues/155

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#18

Post by yevpats » Mon, Jul 24, 2023, 4:44 PM UTC

Congrats!! Check out also CloudQuery (https://github.com/cloudquery/cloudquery) an open source ELT that support DB->DB without Kafka or Debezium (Founder here) :)

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#19

Post by BenderV » Mon, Jul 24, 2023, 4:45 PM UTC

Congrats on your project. Reliable CDC is hard to set up.

I worked a bit on the space, focusing on api instead of DB, which is in the end a quite different problem. My goal was to leverage LLM to auto-build connector on the fly.

I started by building a lightweight ETL, that I open-sourced (https://github.com/BenderV/universal-data).

I left the space because I realize that I didn't want to work on this problem, even though I believe in the "AI" approach, and think simplifying data transfer (or distributing compute) is one the key factor to scale data usage.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#20

Post by jgraettinger1 » Mon, Jul 24, 2023, 5:02 PM UTC

EDEdDNEdDYFaN wrote: post #2

How many of these tools are out there? This is like the 4th CDC to Data warehouse tool I've seen this year and I'm struggling to understand why they all exist in parallel. I just found estuary with a colleague yesterday which didn't work and previously have seen at least two other tools promising the same CDC to Data Warehouse in-a-box pipelines. What makes this tool different?

Hi, I'm Estuary's CTO (https://estuary.dev). Mind speaking a bit more about what didn't work?

We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using them at scale successfully, but they can be a bit nuanced to get configured. We're constantly trying to make our onboarding experience more intuitive and seamless... it's a hard problem.