Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The new local data stack: Integrating Dagster, dbt and DuckDB

georgheiler.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#31

Post by riku_iki »

datadrivenangel wrote: post #27

I've helped with multiple data engineering teams that suffer tremendously due to a lack of local setup, so DuckDB and the right choice of tools looks immensely promising. Also for 99% of companies and data workloads the optimal choice should usually be a janky bash script, so this provides meaningful competition for annoyingly medium data.

> I've helped with multiple data engineering teams that suffer tremendously due to a lack of local setup

why remote setup is not enough for those teams?..

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#32

Post by mritchie712 »

hbarka wrote: post #29

On Dagster’s website there are comparisons to dbt. Is it complementary (as positioned in this stack) or competitive? Dagster versus dbt https://dagster.io/vs Migrating off dbt cloud https://dagster.io/blog/migrate-off-dbt-cloud

you can run dbt core via a dagster job (so it'd replace dbt cloud)

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#33

Post by totalhack »

benjaminwootton wrote: post #7
OJFord wrote: post #5

Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?

There are a bunch of collector tools for capturing application side events and sending them into your server. The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot. Then I think you are referring to what is sometimes called “user facing analytics” against that datastore. We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs…

+1 for using a semantic layer. I built (and open-sourced) one for use in my own business but Cube looks well done.

https://github.com/totalhack/zillion

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#34

Post by mritchie712 »

jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

Definite[0] is the cloud version of this idea (data stack in a box). We have ETL, modeling, a data warehouse (Snowflake), and BI (dashboards) in one app.

We're experimenting with using DuckDB as the warehouse. Would be awesome to let people pull down parts of their warehouse locally for testing.

0 - https://www.definite.app/

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#35

Post by jasmcole »

dotandgtfo wrote: post #23
jasmcole wrote: post #8

If you're interested in interrogating dbt models using DuckDB, you may be interested in some new dbt features we've recently released at https://count.co

Your link probably doesn't lead where you want it to btw.

Thanks for the heads up - a page more focussed on dbt specifically is here:

https://count.co/product/data-modeling/

and associated blog post:

https://count.co/blog/how-we-made-data-modeling-multiplayer/

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#36

Post by jakozaur »

dappermanneke wrote: post #13
jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

what stops you from running this on a container and dumping the results with a script to be queried by a reporting solution into bigquery or snowflake?

I already see companies with BigQuery setuped.

I would like to quickly grab some data, queries and dashboards and run them in my local warehouses.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#37

Post by geoHeil »

dappermanneke wrote: post #13
jakozaur wrote: post #4

It's a cool idea, but it looks incomplete for the production use case. 1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop. 2. Almost all companies already have some data setup. The migration path is very unclear. Mos…

what stops you from running this on a container and dumping the results with a script to be queried by a reporting solution into bigquery or snowflake?

Hi,

Georg - one of the authors is here.

In fact, this is exactly what we argue: Keep data the consumer experience high a Paas platform (Fabric, BigLake, Databricks, SF, ...) can make a lot of sense, whereas for the best data development/creator experience a high-code solution based on solid software engineering best practices should be the preferred solution - at least in my/our(authors) opinion.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#38

Post by geoHeil »

remram wrote: post #21
skypanther wrote: post #14

I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?

It does feel like it was lengthened with GPT. There's so many essay-style "now we propose to show that...", it's uncanny. > we argue that we can rethink the current state of data transformation pipelines (intro) > In the blog we will cover: > We show a way how you can combine the best of both worlds > We will tackle its impact and explain more in the following development section. edit: yeah it's ChatGPT: > Parts of…

Hi,

Georg - one of the authors is here.

Indeed, we used LLMs/GPT4 for proof reading and enhancing the English language (we are not native speakers).

We were thinking about breaking up the content - but decided that one long post is a better fit.

> > We show a way how you can combine the best of both worlds

In fact, that sentence I wrote by hand : )

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#39

Post by geoHeil »

benjaminwootton wrote: post #7
OJFord wrote: post #5

Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?

There are a bunch of collector tools for capturing application side events and sending them into your server. The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot. Then I think you are referring to what is sometimes called “user facing analytics” against that datastore. We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs…

Not to forget https://www.starrocks.io/ or https://risingwave.com/ - see also a recent blog post of a friend: https://juhache.substack.com/p/moving-from-bi-to-data-apps-p...

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#40

Post by geoHeil »

Jgrubb wrote: post #20
geoHeil wrote: post #2

We (Aleksandar and Georg) want to share our new blog post on: "Dagster, dbt, DuckDB as new local MDS" here. It re-introduces the local environment to enhance the developer productivity for data pipelines by bringing back software engineering best practices. We suggest that PaaS platforms should become an implementation detail and refine the new local stack with great a data consumer experience by combining the best o…

Hey Georg, thanks for posting. I've been working on building a thing with almost the same stack lately and the Dagster integration is my next large-ish step. In the last couple weeks I've kicked around Duck and Clickhouse for the backend. It's been a lot of fun. Ultimately I'd like something that can be run locally or easily installed and run on a PaaS. The only trick with local is that data sets of any appreciable s…

Well local can mean your laptop. But could mean your local server. However, it can also mean a VM on your cloud provider of choice colocated to the object store.

In such a case the network transfer (for many cases) can be almost irrelevant - at least for up to medium-ish sized datasets.

Post reply on HN