Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dabbling with Dagster vs. Airflow

davidsj.substack.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Dabbling with Dagster vs. Airflow

#21

Post by sryza »

frankbreetz wrote: post #12

The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset schedu…

Dagster contributor here responding to Astronomer employee ^^.

A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different:

- TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small.

- If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs.

Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.

Re: Dabbling with Dagster vs. Airflow

#22

Post by [deleted] »

valenterry wrote: post #7

Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?

[deleted]

Re: Dabbling with Dagster vs. Airflow

#23

Post by DSJayatillake »

frankbreetz wrote: post #12

The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset schedu…

I don't really think that either will overtake Airflow in terms of sheer number or users or orgs it is deployed at.

1. Dagster and Prefect won't ever have a hosted version on AWS/GCP (teams can spin these up from their consoles without even considering a vendor conversation or leaving their infra).

2. There are very large Airflow projects out there with huge DE teams running them, that just can't really move to Dagster or Prefect very easily. Not do they really want to if they can do what they need to in a tool they are familiar with.

My perspectives have been very much coming at the tools with fresh eyes.

Will they overtake Airflow in terms of features? They probably already have.

Re: Dabbling with Dagster vs. Airflow

#24

Post by lmeyerov »

singhrac wrote: post #19

I recently had the experience of setting up some Prefect pipelines, which I can compare to this article. Note that while I'm not new to data engineering, I'm new to open source frameworks, and have some insight into Airflow (studied architecture in depth, written a lot of code in it). Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks),…

We've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries with care not to overwhelm the targets; 2000's era backpressure good citizen stuff), and found that all the scheduling primitives ended up being half-baked from a data orchestration perspective. That's even before we get into periodic rescoring for the actual AI parts. So these ended up feeling like generic manual task/orchestration tools, which there are simpler + more powerful technologies. We did the same exercise 2-3 years ago, and surprisingly, not much change in core scheduler interfaces here from this perspective.

Curious if there are positive experiences with any tools here from a data orchestration perspective, esp OSS?

Re: Dabbling with Dagster vs. Airflow

#25

Post by KptMarchewa »

sryza wrote: post #21
frankbreetz wrote: post #12

The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset schedu…

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Opera…

Spark, after all _is_ a large Apache project?

Re: Dabbling with Dagster vs. Airflow

#26

Post by skrtskrt »

jmcnevin wrote: post #20
valenterry wrote: post #7

Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?

I'd say give Temporal ( https://temporal.io ) a look, but there are a lot of options ( https://github.com/meirwah/awesome-workflow-engines ).

Seconding Temporal, it is definitely a second-gen workflow platform with approaches that make many Airflow frustrations nonexistent.

In particular, workflows are extremely durable, they can easily just sit indefinitely waiting on a condition and you can redeploy workers whenever, and workflows will resume exactly where they were.

Re: Dabbling with Dagster vs. Airflow

#27

Post by swyx »

lmeyerov wrote: post #24
singhrac wrote: post #19

I recently had the experience of setting up some Prefect pipelines, which I can compare to this article. Note that while I'm not new to data engineering, I'm new to open source frameworks, and have some insight into Airflow (studied architecture in depth, written a lot of code in it). Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks),…

We've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries with care not to overwhelm the targets; 2000's era backpressure good citizen stuff), and found that all the scheduling primitives ended up being half-baked from a data orchestration perspective. That's even before we get into periodic rescoring for the actual AI…

i mean, the leading new OSS solutions are Prefect and Dagster, its not like there are a million of these out there, would just try them out and see what you think, each have their fans

Re: Dabbling with Dagster vs. Airflow

#28

Post by zyd14 »

KptMarchewa wrote: post #25
sryza wrote: post #21

Earlier quoted context omitted.

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Opera…

Spark, after all _is_ a large Apache project?

probably meant databricks

Re: Dabbling with Dagster vs. Airflow

#29

Post by Peteris »

An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.
Post reply on HN