Live data from Hacker News

Dabbling with Dagster vs. Airflow

davidsj.substack.com

31–40 of 56 posts

Re: Dabbling with Dagster vs. Airflow

#31
post #21

Earlier quoted context omitted.

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Opera…

Spark, after all _is_ a large Apache project?

Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors.

Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.

Re: Dabbling with Dagster vs. Airflow

#32

Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?

I'd say give Temporal ( https://temporal.io ) a look, but there are a lot of options ( https://github.com/meirwah/awesome-workflow-engines ).

Wow, that looks cool. Especially since the way of defining things in code looks pretty first-class.

Re: Dabbling with Dagster vs. Airflow

#33
I hear grumbling from a friend on the data engineering side of my company. It takes an enormous amount of effort to stay on top of their data pipelines, and they still have lots of failures in cleaning, transformation, orchestration, and reporting. One product wasn't updated in 5 months !

They've tried everything airflow, informatica, alteryx, etc. They've even built their own custom data flow etl in python.

I often wonder if the real issues they face is more about expectations and standards such centralized logging, easy report/artifact generation, ops management, and hiring more developer oriented data engineers.

Re: Dabbling with Dagster vs. Airflow

#34
post #21

Earlier quoted context omitted.

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Opera…

Spark, after all _is_ a large Apache project?

It is a bit more nuanced.

On one hand, it is immensely successful Apache project.

On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.

There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.

Re: Dabbling with Dagster vs. Airflow

#36
post #27

Earlier quoted context omitted.

We've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries with care not to overwhelm the targets; 2000's era backpressure good citizen stuff), and found that all the scheduling primitives ended up being half-baked from a data orchestration perspective. That's even before we get into periodic rescoring for the actual AI…

i mean, the leading new OSS solutions are Prefect and Dagster, its not like there are a million of these out there, would just try them out and see what you think, each have their fans

We have tried + investigated a variety, trying to avoid unnecessary shade for indiv co's. Hundreds of millions maybe even billions have gone into funding this space now, so folks get twitchy :)

Re: Dabbling with Dagster vs. Airflow

#37

I hear grumbling from a friend on the data engineering side of my company. It takes an enormous amount of effort to stay on top of their data pipelines, and they still have lots of failures in cleaning, transformation, orchestration, and reporting. One product wasn't updated in 5 months ! They've tried everything airflow, informatica, alteryx, etc. They've even built their own custom data flow etl in python. I often…

Well it all starts with seductive but mostly bullshit idea that once all the data is transformed and put in some common data lake some amazing, deep insights can be extracted which were not possible before. And there is large budget to be spent on developers, hardware, software, consulting and so on. Who will say no to this?

Re: Dabbling with Dagster vs. Airflow

#39
I tried (and tried, and tried, and tried) once to set up a local "test" instance of Airflow just to try out a few different things and understand better how the whole thing worked. I finally gave up after a week - I've never come across any software that I couldn't install, but Airflow just ended up being too much.

Re: Dabbling with Dagster vs. Airflow

#40

I hear grumbling from a friend on the data engineering side of my company. It takes an enormous amount of effort to stay on top of their data pipelines, and they still have lots of failures in cleaning, transformation, orchestration, and reporting. One product wasn't updated in 5 months ! They've tried everything airflow, informatica, alteryx, etc. They've even built their own custom data flow etl in python. I often…

I think a partial, but substantive explanation is related to fragmented domain expertise across the stakeholder base and lack of ownership of the vision of what the stack is meant to achieve. The recent blog post from the Lago folks about why they exited the no-code reverse ETL space discusses parts of this problem [1].

> Marketers often already have access to data, at least in ‘read-only’ mode, and can download it in CSV format. They don’t really explore their options, not because they would need to learn SQL (no need with spreadsheets), but because this would require them to study the whole data structure of the company.

A database usually contains dozens or hundreds of data tables, and understanding how they are organized, how they relate to each other and how often they are updated is a huge effort. Therefore, they say they want to be more data-driven, but they rarely acquire the knowledge that is required to do so, because the bar is pretty high and because this would add up to their existing workload

The inverse observation here I believe is also relevant and rings true. Engineers often don't have have the marketing/sales/analytics/business domain context to make decisions that inform optimal infrastructure decisions. There's a bidirectional dog-piling effect that just gets worse over time.

My own personal experience has led me to believe that this is largely an ownership and governance issue. It's not enough to just give marketers access to the data warehouse, and it's also not enough to sync up with engineers about KPIs and OKRs on a weekly basis. The result of that is the equivalent of a kids soccer game where everyone is chasing the ball rather than playing their positions as a team.

Adding tooling complexity on top of all that, pressed into the service of fixing "pipeline woes", compounds the problem substantially.

[1] https://www.getlago.com/blog/why-we-moved-away-from-the-no-c...

Post reply on HN