(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…
Launch HN: Artie (YC S23) – Real time data replication to data warehouses
41–50 of 58 posts
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#42Earlier quoted context omitted.
I think you missed the parent's point - your USP is real-time replication. So everything you're proposing makes it not real time. Your USP is now worthless (in that context) and you're competitors are numerous.
Hm, perhaps I wasn't being clear, apologies for that. What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#43The core of the answer was that cloud warehouses are not suitable for real-time use cases, because the batch processing and transformations take too long. If you want real-time, you need to pay up - hence Databricks / Spark. I did call out the fraud use case in that answer.
There were 1st generation ETL tools like Alooma that tried to go into the direction of streaming, and they pushed the limits.
Back then (we had built cloud warehouse monitoring tool), the closest to real-time I've ever seen any company get was IronSource. The time between an event and until that event was available in a dashboard was five minutes (they were using Redshift).
I'll stick my neck out and say that certain industries will be all over Artie, whereas others will shrug their shoulders.
The industries that I think will be all over Artie:
- FinTech - Insurance - AdTech - Gaming - Publishing - Logistics / Delivery
These are industries where a couple of minutes of difference in data recency can make a difference of millions of dollars. And you'll probably cost less than existing streaming solutions, which is obviously nice. But I think the real advantage will be simplicity.
I know you can't support all destinations at once, and need to go with where demand is. But I would expect that the Materialize and Clickhouse crowds are good target users for you.
Good luck, this is an exciting product!
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#44Earlier quoted context omitted.
Hm, perhaps I wasn't being clear, apologies for that. What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.
Huh? The parent's point was your underlying dataset is always in real-time. There's no issue querying a data warehouse when all you're doing is looking for a simple transactional report.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#45Will give artie a go
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#46Very cool, are you guys focused on databases as sources or do you plan to add API-based sources too (e.g.. Stripe Customers)? Currently using Airbyte but something more real-time would be beneficial to us.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#47Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#48For those using Elixir and who want to work with Postgres CDC at a lower level, I've put together a little CDC library: https://github.com/cpursley/walex
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#49Hi Robin, This looks like very cool project Note thought Elastic Licence 2.0 is not Open Source License so if this is your license choice I'd avoid calling the code you have on gitHub OSS