The blocker for testing this out and using it at my place of employment is lack of binary data type in MySQL, any plans to add this?
Launch HN: Artie (YC S23) – Real time data replication to data warehouses
21–30 of 58 posts
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#22The blocker for testing this out and using it at my place of employment is lack of binary data type in MySQL, any plans to add this?
Definitely. Do you expect the resulting data to also be in binary / bytes format in your DWH?
I ask because there's a workaround by setting `binary.handling.mode` to a STRING type [1].
Transfer will then automatically pick this up and write this as a B64 string to the DWH.
[1] https://debezium.io/documentation/reference/stable/connector...
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#23Congrats!! Check out also CloudQuery ( https://github.com/cloudquery/cloudquery ) an open source ELT that support DB->DB without Kafka or Debezium (Founder here) :)
Thank you!!
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#24Very cool, are you guys focused on databases as sources or do you plan to add API-based sources too (e.g.. Stripe Customers)? Currently using Airbyte but something more real-time would be beneficial to us.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#25Hey, brief feedback, but "5,000,000,000+" rows processed might be a red flag for some. Many individual customers might do this per day, so to say that's all the company has done so far might put them off. You've said that initial imports are free, but anecdotally, initial imports ended up being a somewhat regular occurrence as we found issues required a re-import, table rewrites, breaking schema changes, and so on. D…
> Hey, brief feedback, but "5,000,000,000+" rows processed might be a red flag for some. Many individual customers might do this per day, so to say that's all the company has done so far might put them off. Appreciate the feedback! Totally agree and it's something we are working towards :). We learned a ton from going from nothing -> billions, certain functions, backfill strategies, etc just doesn't work at that scal…
I just wanted to second the GP's point on pricing - like them, "call me pricing" is a hard no from me. IMO it's fine for the highest tier (Enterprise or whatever), but lower tiers should have fixed and public pricing.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#26(None of the below is meant to diminish the work done by the author/poster.)
> This means that when companies aggregate production data into their data warehouse, the underlying data is always stale.
This is intentional and desirable.
The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-...
Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed every 6 hours (limited by upstream), that eventually Marketing asked me to dial back to every 24 hours because they couldn't stand things changing in the middle of the day.
Now of course I see you're targeting more real-time use cases like fraud detection. That's great! But why you would run a fraud detection process out of your data warehouse, which likely doesn't even have a production-grade uptime SLA? Run it out of your production database, that's what it's for!
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#27The blocker for testing this out and using it at my place of employment is lack of binary data type in MySQL, any plans to add this?
Definitely. Do you expect the resulting data to also be in binary / bytes format in your DWH? I ask because there's a workaround by setting `binary.handling.mode` to a STRING type [1]. Transfer will then automatically pick this up and write this as a B64 string to the DWH. [1] https://debezium.io/documentation/reference/stable/connector...
Thanks for the reply! We'll be keeping an eye on it
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#28Very cool, are you guys focused on databases as sources or do you plan to add API-based sources too (e.g.. Stripe Customers)? Currently using Airbyte but something more real-time would be beneficial to us.
For now, we're super focused on databases as sources. We really want to do this well before we move on to other data sources such as APIs.
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#29(None of the below is meant to diminish the work done by the author/poster.) > This means that when companies aggregate production data into their data warehouse, the underlying data is always stale. This is intentional and desirable. The classic piece on this is this one by Dan McKinley https://mcfunley.com/whom-the-gods-would-destroy-they-first-... Something McKinley doesn't address is that it's quite advantageous…
Thanks for your feedback!
> Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed every 6 hours (limited by upstream), that eventually Marketing asked me to dial back to every 24 hours because they couldn't stand things changing in the middle of the day.
If they want to see a consistent view of the report, you could bound this.
1/ SELECT * FROM FOO WHERE DATE_TRUNC('day', updated_at) If your dataset doesn't contain kv, you can turn on include `artie_updated_at` which will provide an additional column with the updated_at field to support incremental ingestion.
2/ If you had stateful data, you could also explore creating a Snowflake task and leveraging the time travel f(x) to create a "snapshot" if your workload depended on it.
3/ Also, if you _did_ want this to be more lagged, you can actually increase the flushIntervalSeconds [1] to 6h, 24h, whichever time interval you fancy. You as the customer should have maximum flexibility when it comes to when to flush to DWH.
4/ You can also choose to refresh the analytical report on Looker / Mode to be daily. [2]
> Now of course I see you're targeting more real-time use cases like fraud detection. That's great! But why you would run a fraud detection process out of your data warehouse, which likely doesn't even have a production-grade uptime SLA? Run it out of your production database, that's what it's for!
You can certainly do this in production db (that was our original hypothesis as well!), however, after talking to more companies...it has become more obvious to us that folks that are running fraud algos actually want to join this across various data sets. Further, by using a DWH - it provides a nice visualization layer on top.
Of course, you could go with something even more bespoke by utilizing real-time DBs such as Materialize / Rockset / RisingWave. Just comes with trade offs such as increase in architectural complexity.
There are also plenty of additional use cases this can unlock given that DWH is a platform, any post-DWH application can benefit from less lag, such as reverse ETLs.
[1] https://docs.artie.so/running-transfer/options
[2] https://mode.com/help/articles/report-scheduling-and-sharing...
Re: Launch HN: Artie (YC S23) – Real time data replication to data warehouses
#30Very cool, are you guys focused on databases as sources or do you plan to add API-based sources too (e.g.. Stripe Customers)? Currently using Airbyte but something more real-time would be beneficial to us.
estuary.dev may be a fit for you (am CTO).
(Competitive product and I feel weird about replying in Artie's thread, but tang8330 has said they're not serving this segment)