Dabbling with Dagster vs. Airflow
davidsj.substack.com
Dabbling with Dagster vs. Airflow
1–10 of 56 posts
Re: Dabbling with Dagster vs. Airflow
#2David Jayatillake's take on Dagster vs. the problems he had in Airflow.
Re: Dabbling with Dagster vs. Airflow
#3sounds like a very positive experience all around, but not going too deep into dagster itself yet. i feel like comparing MWAA to Dagster is not an even footing, i'd be interested in seeing how Astronomer has also improved the Airflow experience.
Re: Dabbling with Dagster vs. Airflow
#4Dagster's product is great, but comparing it to MWAA is unfair. MWAA is a poor quality product - difficult to use, unstable, inflexible, and poorly-supported. A fairer comparison would be against Astronomer. Astronomer is a MUCH better product than MWAA.
Re: Dabbling with Dagster vs. Airflow
#5Dagster's product is great, but comparing it to MWAA is unfair. MWAA is a poor quality product - difficult to use, unstable, inflexible, and poorly-supported. A fairer comparison would be against Astronomer. Astronomer is a MUCH better product than MWAA.
[deleted]
Re: Dabbling with Dagster vs. Airflow
#6Dagster's product is great, but comparing it to MWAA is unfair. MWAA is a poor quality product - difficult to use, unstable, inflexible, and poorly-supported. A fairer comparison would be against Astronomer. Astronomer is a MUCH better product than MWAA.
Any pointers to Astronomer vs. Dagster comparisons or just good experience reports with Astronomer?
Re: Dabbling with Dagster vs. Airflow
#7Where is a service that doesn't focus specifically on
1.) data pipelines / data-science
2.) cicd / build pipelines
3.) ... you name it
I mean, just a service that gives me all the groundwork to build one of the above myself.
Is there something like that?
Re: Dabbling with Dagster vs. Airflow
#8No post body was provided.
Re: Dabbling with Dagster vs. Airflow
#9Dagster's product is great, but comparing it to MWAA is unfair. MWAA is a poor quality product - difficult to use, unstable, inflexible, and poorly-supported. A fairer comparison would be against Astronomer. Astronomer is a MUCH better product than MWAA.
Any pointers to Astronomer vs. Dagster comparisons or just good experience reports with Astronomer?
I don't know one off the top of my head, but here are a few bullet points:
* Both provide fully-managed or hybrid SAAS options. Dagster has Dagster Cloud, while Astronomer has Astro.
* Both are containerized deployments.
* Both have fully-functional local development environments. My Airflow development environment is my local env managed by the Astro CLI. It works great. I haven't worked with Dagster, but I've heard that it's lighter weight and local development is delightful.
* Both have functionality for data lineage. Dagster has Software Defined Assets, while Airflow/Astronomer integrates with OpenLineage. I don't have direct experience here, btw.
* Both are reasonably priced. They're cheap enough that the build vs buy decision is a no-brainer: buy buy buy.
One difference, however, is that Dagster natively isolates tasks in the DAG into separate Kubernetes pods. You can get this with Airflow if you use KubernetesExecutor, but KubernetesExecutor is only available in Astronomer's Hybrid-SAAS product. Only CeleryExecutor is available in Astro.
There's also a difference in syntax for defining DAGs. I find Dagster's approach is more Pythonic than Airflow's standard way of defining DAGs. However, Airflow's TaskFlow API is almost at parity with Dagster's approach.
One advantage that Airflow has over Dagster is its maturity and pre-built integrations with other systems. If I need to interact with Fivetran in Airflow, there's a provider for that. Hightouch? there's a provider. Snowflake? Yep. If there's a widely used product or managed service related to data, there's probably an Airflow provider for it.
Re: Dabbling with Dagster vs. Airflow
#10Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?
Shameless self-promotion: I built Eventline (see my profile) precisely because I just wanted to schedule and run code in various environments, manually or in reaction to various events, without the complexity that goes with specialized tools.
It gives you a pretty flexible job system, identities (secrets), monitoring, notifications, a web interface, a command line tool and a HTTP API. You can build lots of things above it.
Do you have specific use cases in mind ?