Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The new local data stack: Integrating Dagster, dbt and DuckDB

georgheiler.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#21

Post by remram »

skypanther wrote: post #14

I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?

It does feel like it was lengthened with GPT. There's so many essay-style "now we propose to show that...", it's uncanny.

> we argue that we can rethink the current state of data transformation pipelines (intro)

> In the blog we will cover:

> We show a way how you can combine the best of both worlds

> We will tackle its impact and explain more in the following development section.

edit: yeah it's ChatGPT:

> Parts of this text were adeptly generated by ChatGPT but enhanced by real humans.

I guess it's the future, turn a tweet into a 5-page essay with your AI, so your readers can summarize it back to tweet-length with their AI.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#24

Post by bart_spoon »

skypanther wrote: post #14

I only skimmed the article due to its length. But I didn't see anything like a comparison to other toolchains. Like, how is this better/different/worse than Airflow + dbt + Snowflake?

Far more helpful to me than these essays on the benefits of a particular paradigm is a simple, minimal example in a repo that I can dig into and explore.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#25

Post by bart_spoon »

Eumenes wrote: post #10

Looks cool but I don't like to push to paid managed services like dbt. Rather use something like Airflow but its not as easy to use for most teams.

DBT is open source and can be used entirely for free. They sell a hosted version.

Also DBT and Airflow are different tools, aimed at solving completely different problems. The Airflow competitor in this article is Dagster, which is also open source.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#26

Post by gms »

OJFord wrote: post #5

Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data. What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?

For the ingesting bit, we (Polytomic) have an API you can use: https://www.polytomic.com/connect.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#27

Post by datadrivenangel »

I've helped with multiple data engineering teams that suffer tremendously due to a lack of local setup, so DuckDB and the right choice of tools looks immensely promising.

Also for 99% of companies and data workloads the optimal choice should usually be a janky bash script, so this provides meaningful competition for annoyingly medium data.

Re: The new local data stack: Integrating Dagster, dbt and DuckDB

#30

Post by jnac »

hbarka wrote: post #29

On Dagster’s website there are comparisons to dbt. Is it complementary (as positioned in this stack) or competitive? Dagster versus dbt https://dagster.io/vs Migrating off dbt cloud https://dagster.io/blog/migrate-off-dbt-cloud

Dagster (orchestration tool) is complementary to dbt core (open source CLI tool for defining data transformation jobs) but competitive to dbt cloud (SaaS, orchestrates dbt jobs).
Post reply on HN