Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Airflow's Problem

#111

Post by kumare3 »

glogla wrote: post #91
shmoogy wrote: post #65

Earlier quoted context omitted.

I've definitely noticed more issues after adding users, but it's more that they don't actually understand a lot of what they're trying to do and cause problems when writing dags.

Yeah, Airflow isn't multi-tenant. People can potentially overwrite each other's DAGs. Credential management is complicated. Broken DAG can stop whole Airflow. Slow DAG can impact performance of whole Airflow. Getting DAGs to wait for each other (like one team prepares data up to a point and then other team builds on that) is kind of a nightmare. Sometimes people want features from newer Airflow, but some other team b…

Some of these were the core problems that we wanted to address as part of https://flyte.org. We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are restricted to teams, tasks and workflows are shareable across teams, without making libraries. and it is possible to trigger workflows across teams. Each teams workflows are tasks and grouped using a construct called projects. It is even possible to separate execution clusters per team, per workflow onto separate k8s clusters. Also the platform is built to be managed and easily deployed.

Re: Airflow's Problem

#112

Post by metadat »

code_biologist wrote: post #88
zibarn wrote: post #61

Earlier quoted context omitted.

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

An analogy is "can you tell what problems Django solves that can't be handled by wsgi and psycopg?" Nothing fundamentally different, but life is a whole lot easier with Django. Honestly if you're doing data engineering and you haven't spent time with a good DAG runner, you're doing yourself a real disservice. My sibling comment did a good job explaining, but the UI + configurable storage + configurable triggers all o…

Django is easier when you want to do things only the "Django" way. However once you need something done differently it quickly shows its truly rigid and brittle self, and you'll find yourself fighting a great and challenging battle.

Re: Airflow's Problem

#113

Post by sspaeti »

Fascinating article and to the heart of my being as a DE engineer. I had the same conclusion a while back and wrote about the trends in data orchestration towards a declarative pipeline, which more modern tools, as you mentioned, will do, but I'm afraid not Airflow.

I'd say data consumers, such as data analysts and business users, care primarily about the production of data assets. On the other hand, data engineers with Airflow focus on modeling the dependencies between tasks (instead of data assets). How how can we reconcile both worlds?

In my latest article, I review Airflow, Prefect, and Dagster and discuss how data orchestration tools introduce data assets as first-class objects. I also cover why a declarative approach with higher-level abstractions helps with faster developer cycles, stability, and a better understanding of what’s going on pre-runtime. I explore five different abstractions (jobs, tasks, resources, triggers, and data products) and see if it all helps to build a Data Mesh. If that sounds interesting, make sure to check out https://airbyte.com/blog/data-orchestration-trends.

Re: Airflow's Problem

#114

Post by orthoxerox »

coldtea wrote: post #81
orthoxerox wrote: post #62

A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS…

> and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. https://news.ycombinator.com/item?id=9224 > Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. I guess that's one approach to job security. And why no…

I think you've misunderstood my point. I don't want all these problems to be mine, I want whatever job orchestrator I choose to solve them. Airflow very explicitly doesn't try to solve them, doesn't even try to solve them badly, it just runs jobs when scheduled.

Re: Airflow's Problem

#115

Post by orthoxerox »

robertlagrant wrote: post #94
orthoxerox wrote: post #62

A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS…

> isn't Airflow's problem, it's your problem This is a baffling statement.

It is, that's why I was so baffled by the feature set of Airflow. It doesn't even try to take a stab at these problems.

Re: Airflow's Problem

#116

Post by coldtea »

orthoxerox wrote: post #114
coldtea wrote: post #81

Earlier quoted context omitted.

> and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. https://news.ycombinator.com/item?id=9224 > Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. I guess that's one approach to job security. And why no…

I think you've misunderstood my point. I don't want all these problems to be mine, I want whatever job orchestrator I choose to solve them. Airflow very explicitly doesn't try to solve them, doesn't even try to solve them badly, it just runs jobs when scheduled.

Well, Airflow is not that great, but does try to solve these problems: it has retries, it shows "jobs are on the critical path" for another job, and how they're doing, and so on...

Re: Airflow's Problem

#117

Post by shrikant »

metadat wrote: post #112
code_biologist wrote: post #88

Earlier quoted context omitted.

An analogy is "can you tell what problems Django solves that can't be handled by wsgi and psycopg?" Nothing fundamentally different, but life is a whole lot easier with Django. Honestly if you're doing data engineering and you haven't spent time with a good DAG runner, you're doing yourself a real disservice. My sibling comment did a good job explaining, but the UI + configurable storage + configurable triggers all o…

Django is easier when you want to do things only the "Django" way. However once you need something done differently it quickly shows its truly rigid and brittle self, and you'll find yourself fighting a great and challenging battle.

Perhaps unwittingly, you've hit upon people's exact frustrations with Airflow! :)

Re: Airflow's Problem

#120

Post by robertlagrant »

orthoxerox wrote: post #115
robertlagrant wrote: post #94

Earlier quoted context omitted.

> isn't Airflow's problem, it's your problem This is a baffling statement.

It is, that's why I was so baffled by the feature set of Airflow. It doesn't even try to take a stab at these problems.

Oh! I took that to mean that you thought Airflow shouldn't do it, not that Airflow thought Airflow shouldn't do it.
Post reply on HN