Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The new local data stack: Integrating Dagster, dbt and DuckDB

georgheiler.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#41

Post by geoHeil »

jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

Hi,

Georg - one of the authors is here.

What we argue is that:

- for a great software/data engineering/creation experience we recommend such a stack that is only on when needed (when transformations occur)

- for a great data consumption experience we suggest the integration with an established PaaS platform. Not only for the sake of being available (as a serving layer of data to end users) but also for the missing fine-grained RBAC in the proposed transformation layer

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#42

Post by geysersam »

jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

Isn't that case easily handled by something like a large stateless VM and cloud storage, so EC2 + S3 ? Doesn't have to be local, the point is that it doesn't have to be distributed either, just one large instance that is only on when it's needed.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#43

Post by dotandgtfo »

jasmcole wrote: post #35
dotandgtfo wrote: post #23

Earlier quoted context omitted.

Your link probably doesn't lead where you want it to btw.

Thanks for the heads up - a page more focussed on dbt specifically is here: https://count.co/product/data-modeling/ and associated blog post: https://count.co/blog/how-we-made-data-modeling-multiplayer/

Thanks! I was logged into count so I just got sent straight to my homepage as well. Seems interesting

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#44

Post by mgaunard »

wenc wrote: post #28
mgaunard wrote: post #16

I use DuckDB, it's a great piece of tech. Only way I could figure out how to use it though was to read the source (found a few interesting bugs while doing so as well)

No code is fully free of bugs. Did you submit an issue or PR?

I didn't do all of them because I found them on the job.

Submitted one bug report for the thing I didn't know how to fix.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#45

Post by geekodour »

kalendos wrote: post #22

I've been working with this stack (building Open Data Portals¹) for a few months and am super happy with how well everything plays together. ¹ https://github.com/davidgasquez/gitcoin-grants-data-portal

Hi, i am working on something similar and was looking for ways how i can host my open data. the approach seems interesting, can i reach out to you to discuss more on this somewhere?

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#46

Post by adammarples »

skypanther wrote: post #14

I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?

It just seems like one choice, among many. The dagster-dbt library in particular seems like a slightly pointless wrapper around the dbt cli and the json artifacts it creates. I've been using airflow and astronomer-cosmos which is not perfect but I didn't enjoy using dagster last time I tried.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#47

Post by geoHeil »

jakozaur wrote: post #36
dappermanneke wrote: post #13

Earlier quoted context omitted.

what stops you from running this on a container and dumping the results with a script to be queried by a reporting solution into bigquery or snowflake?

I already see companies with BigQuery setuped. I would like to quickly grab some data, queries and dashboards and run them in my local warehouses.

If you use bigquery in big lake mode - i.e. with parquet or iceberg or delta files on GCS (or other object store) you can easily pull in the data into duckdb as well

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#48

Post by geoHeil »

adammarples wrote: post #46
skypanther wrote: post #14

I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?

It just seems like one choice, among many. The dagster-dbt library in particular seems like a slightly pointless wrapper around the dbt cli and the json artifacts it creates. I've been using airflow and astronomer-cosmos which is not perfect but I didn't enjoy using dagster last time I tried.

Interesting why? What were the pain points?

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#49

Post by kalendos »

geekodour wrote: post #45
kalendos wrote: post #22

I've been working with this stack (building Open Data Portals¹) for a few months and am super happy with how well everything plays together. ¹ https://github.com/davidgasquez/gitcoin-grants-data-portal

Hi, i am working on something similar and was looking for ways how i can host my open data. the approach seems interesting, can i reach out to you to discuss more on this somewhere?

Sure! You can find my contact details on GitHub: https://github.com/davidgasquez.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#50

Post by milicevica23 »

jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

Hi, Aleks here, one of the authors, and thank you very much for your comment

We run this stack in production for the last few months, and it has its downsides (I would argue due to the young ecosystem) and upsides, which we try to explain in the blog. We wanted to concentrate more on the concepts and technology change/improvement that allows us to run such a stack and explain how we see the future steps forward.

1. Running a warehouse is not a bad idea, but you must always be careful to separate the storage from the compute to scale. I experienced the limitation of such a system as described in this blog https://delta.io/blog/2022-09-14-why-migrate-lakehouse-delta... -> tough position if your solution is good but not scale. Ideally run it with external tables in order that data is visible without engine access

Another limitation is the metastore for your tables and metadata, which you usually have per workspace/environment in such a scenario. Databricks' unity catalog is an excellent way to solve it, but it is only compatible with some engines.

2. We do not think that this stack exists to exchange the snowflake or big query but to take a part of the workload away ( data transformation) and let PaaS solutions be good at what they are made for -> user interface and interaction.

Post reply on HN