Live data from Hacker News

Launch HN: Artie (YC S23) – Real time data replication to data warehouses

news.ycombinator.com

41–50 of 58 posts

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#41

(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…

Snowflake and Big Query support as-of queries. Redshift supports snapshots. One could easily get a report that's consistent over the course of time.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#42
post #31

Earlier quoted context omitted.

I think you missed the parent's point - your USP is real-time replication. So everything you're proposing makes it not real time. Your USP is now worthless (in that context) and you're competitors are numerous.

Hm, perhaps I wasn't being clear, apologies for that. What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.

Huh? The parent's point was your underlying dataset is always in real-time. There's no issue querying a data warehouse when all you're doing is looking for a simple transactional report.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#43
In 2016, I wrote a pretty detailed answer for "Spark vs. Redshift" question. This was in the very early days of what today I guess is called "the modern data stack"

The core of the answer was that cloud warehouses are not suitable for real-time use cases, because the batch processing and transformations take too long. If you want real-time, you need to pay up - hence Databricks / Spark. I did call out the fraud use case in that answer.

There were 1st generation ETL tools like Alooma that tried to go into the direction of streaming, and they pushed the limits.

Back then (we had built cloud warehouse monitoring tool), the closest to real-time I've ever seen any company get was IronSource. The time between an event and until that event was available in a dashboard was five minutes (they were using Redshift).

I'll stick my neck out and say that certain industries will be all over Artie, whereas others will shrug their shoulders.

The industries that I think will be all over Artie:

- FinTech - Insurance - AdTech - Gaming - Publishing - Logistics / Delivery

These are industries where a couple of minutes of difference in data recency can make a difference of millions of dollars. And you'll probably cost less than existing streaming solutions, which is obviously nice. But I think the real advantage will be simplicity.

I know you can't support all destinations at once, and need to go with where demand is. But I would expect that the Materialize and Clickhouse crowds are good target users for you.

Good luck, this is an exciting product!

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#44
post #42

Earlier quoted context omitted.

Hm, perhaps I wasn't being clear, apologies for that. What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.

Huh? The parent's point was your underlying dataset is always in real-time. There's no issue querying a data warehouse when all you're doing is looking for a simple transactional report.

I think their point is they have a real-time warehouse that can also be used in “stale snapshot” mode.

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#46
post #24

Very cool, are you guys focused on databases as sources or do you plan to add API-based sources too (e.g.. Stripe Customers)? Currently using Airbyte but something more real-time would be beneficial to us.

Take a look at Nango.dev

Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses

#49

Hi Robin, This looks like very cool project Note thought Elastic Licence 2.0 is not Open Source License so if this is your license choice I'd avoid calling the code you have on gitHub OSS

You mean AWS can't deliver this as a service so it's not open source? That's an interpretation and not gospel
Post reply on HN