Looks cool but I don't like to push to paid managed services like dbt. Rather use something like Airflow but its not as easy to use for most teams.
I'm not sure those two solve the same problem. Regardless airflow is an amazingly useful tool.
The new local data stack: Integrating Dagster, dbt and DuckDB
11–20 of 52 posts
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#12Looks cool but I don't like to push to paid managed services like dbt. Rather use something like Airflow but its not as easy to use for most teams.
dbt-core is open source and can be used locally.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#13It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…
what stops you from running this on a container and dumping the results with a script to be queried by a reporting solution into bigquery or snowflake?
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#14I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#15It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…
I could add more but on a newish local stack I was trying: I spent a good amount of hours on duckdb this week to process personal data from data dumps (social networks, etc) and now I'm back to the combo of postgresql in containers + sqlite.
After the initial imports and some massaging with queries that felt awesome, I found hard to step-up the game building the relationships I wanted. The last stroke before switching was the lack of managing foreign keys without recreating entire tables again. I can go over other examples.
It can be done, but it just takes you out of the flow when you're analyzing the data + cleaning it, specially because I know that I can do with psql and sqlite in a blink of an eye.
Since many etl tools don't care about the target database being these old and trusty fellas, I felt losing a lot of time just to get rid of a postgres install that is right now consuming only 200 mb of ram on a docker/podman container. Or working around some sqlite ingestion issues with simple notebooks + pandas/polars/etc.
in my pov it seems a shaky ground for an entire new stack
I appreciate duckdb taking me out of the comfort zone tho.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#16I use DuckDB, it's a great piece of tech. Only way I could figure out how to use it though was to read the source (found a few interesting bugs while doing so as well)
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#17I use DuckDB, it's a great piece of tech. Only way I could figure out how to use it though was to read the source (found a few interesting bugs while doing so as well)
Which bug?
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#18I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?
Snowflake isn't local, you have to pay for cloud ... Airflow is Airflow, complexity, steep learning curve, there is a whole industry trying to be as/more powerful than Airflow with less complexity and cleaner integration with modern things you might want to do like k8s, although of course Airflow is still super popular and powerful.
the Dagster folks have some comparisons, of course there are popular modern alternatives other than Dagster
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#19Looks cool but I don't like to push to paid managed services like dbt. Rather use something like Airflow but its not as easy to use for most teams.
As the other poster mentioned, DBT is an open core product and I'm not sure how you'd use the cloud service on a local DB install anyway. Airflow and DBT don't really do the same things anyway.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#20We (Aleksandar and Georg) want to share our new blog post on: "Dagster, dbt, DuckDB as new local MDS" here. It re-introduces the local environment to enhance the developer productivity for data pipelines by bringing back software engineering best practices. We suggest that PaaS platforms should become an implementation detail and refine the new local stack with great a data consumer experience by combining the best o…
Hey Georg, thanks for posting. I've been working on building a thing with almost the same stack lately and the Dagster integration is my next large-ish step. In the last couple weeks I've kicked around Duck and Clickhouse for the backend. It's been a lot of fun. Ultimately I'd like something that can be run locally or easily installed and run on a PaaS.
The only trick with local is that data sets of any appreciable size take ages to pull down, at least here in the US with our terrible internet (in the average Italian mountain village this would probably work great).