The new local data stack: Integrating Dagster, dbt and DuckDB
1–10 of 52 posts
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#2We (Aleksandar and Georg) want to share our new blog post on: "Dagster, dbt, DuckDB as new local MDS" here.
It re-introduces the local environment to enhance the developer productivity for data pipelines by bringing back software engineering best practices.
We suggest that PaaS platforms should become an implementation detail and refine the new local stack with great a data consumer experience by combining the best of both worlds.
Please take a read here: https://georgheiler.com/2023/12/11/dagster-dbt-duckdb-as-new...
We (Georg Heiler and Aleksandar Milicevic) are keen to discuss the proposed new stack with you. Do not hesitate to reach out.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#3We (Aleksandar and Georg) want to share our new blog post on: "Dagster, dbt, DuckDB as new local MDS" here. It re-introduces the local environment to enhance the developer productivity for data pipelines by bringing back software engineering best practices. We suggest that PaaS platforms should become an implementation detail and refine the new local stack with great a data consumer experience by combining the best o…
[deleted]
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#4It's a cool idea, but it looks incomplete for the production use case.
1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop.
2. Almost all companies already have some data setup. The migration path is very unclear. Most likely, this is a secondary system for the majority of companies. Ideally, I would love to describe how I can use it along big platforms (e.g. BigQuery or Snowflake).
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#5Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data.
What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#6Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?
Bunch of python code.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#7Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?
There are a bunch of collector tools for capturing application side events and sending them into your server.
The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot.
Then I think you are referring to what is sometimes called “user facing analytics” against that datastore.
We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs and a semantic layer.
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#8If you're interested in interrogating dbt models using DuckDB, you may be interested in some new dbt features we've recently released at https://count.co
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#9Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?
There are a bunch of collector tools for capturing application side events and sending them into your server. The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot. Then I think you are referring to what is sometimes called “user facing analytics” against that datastore. We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs…
'user-facing analytics' and (via Cube.dev) 'embedded analytics' are helpful terms, thank you!
Re: The new local data stack: Integrating Dagster, dbt and DuckDB
#10Looks cool but I don't like to push to paid managed services like dbt. Rather use something like Airflow but its not as easy to use for most teams.