Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: Airflow's Problem

#81

Post by coldtea »

orthoxerox wrote: post #62

A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS…

>and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure.

https://news.ycombinator.com/item?id=9224

>Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem.

I guess that's one approach to job security. And why not make data egress manual too? Why transfer data through the network, when you can print them, mail the papers, and type them back in? Data input is not the computer's problem, it's your problem!

>Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. "What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem. "Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem.

The whole idea of writing programs is making things automatable. That is, making them the computer's problem, not our problem. We get the higher level problem of writing the automation once, and fixing any bugs in our code, then we get to enjoy putting it to work for us...

Re: Airflow's Problem

#82

Post by bushbaba »

Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

The idea behind airflow is great. What sucks is people using it to do heavy processing. Maybe with serverless/k8s airflow could fan out the processing to a cluster to allow for flexibility. But then, I guess you end up re-writing spark et-al.

Re: Airflow's Problem

#83

Post by rldjbpin »

Having been forced to work with an obsolete version of Airflow at work, I can attest to how narrow-minded the project's focus was when it was originally created. The scheduling quirks and UTC defaults are enough to paint the picture here.

Not completely sure if most of the issues I've faced were resolved in the future releases, but I don't fully agree with the take of the article. Like go with the scheduler that works for your current and potential future needs. The reason why we continue to use Airflow despite the issues is because it works so well with our workflows. This does mean that I would recommend it to another team.

Re: Airflow's Problem

#84

Post by claytonjy »

datascientist wrote: post #78

Recent perspectives from the creators of Prefect, Dagster, Flyte, and Orchest => https://gradientflow.com/summer-of-orchestation/

Seems to be missing Temporal/Cadence, which I'm very excited about, but I've never heard of Flyte or Orchest.

Re: Airflow's Problem

#85

Post by jdoliner »

I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.

[0] https://github.com/pachyderm/pachyderm

Re: Airflow's Problem

#86

Post by systemvoltage »

mumblemumble wrote: post #44
llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt. It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a who…

Yet, we abide by data-oriented conclusions outside of software engineering all the time. From Academics papers to FDA to crime statistics.

Re: Airflow's Problem

#87

Post by calebm »

> If it sounds like you could simply replace Airflow with basically any other job execution engine, that’s because you could.

Has anyone tried Luigi for data engineering pipelines?

Re: Airflow's Problem

#88

Post by code_biologist »

zibarn wrote: post #61
Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

An analogy is "can you tell what problems Django solves that can't be handled by wsgi and psycopg?" Nothing fundamentally different, but life is a whole lot easier with Django. Honestly if you're doing data engineering and you haven't spent time with a good DAG runner, you're doing yourself a real disservice.

My sibling comment did a good job explaining, but the UI + configurable storage + configurable triggers all out of the box make life a lot easier.

Re: Airflow's Problem

#89

Post by pdinny »

evrydayhustling wrote: post #20
pdinny wrote: post #12

The post feels like a bait-and-switch in the sense that it presents itself as about Airflow's shortcomings but focuses mostly on problems that Airflow doesn't attempt to solve. Airflow can certainly be frustrating and it doesn't solve _all_ workflow orchestration problems. Surely the same thing can be said of many tools? This seems mostly like a mismatch of expectations.

FWIW the author is pretty direct about this. After the cute beginning, he basically says that his problem is with Airflow's scope, not it's execution.

Hence the bait and switch. IMHO increasing the scope of Airflow (or any tool) is a challenging proposition. Would you rather use very few mega-tools with very broad scope (and potentially more challenging domain to navigate) or fewer more specialised tools that interoperate well together?

Obviously there are trade-offs with either approach, but then I'd argue that making Airflow solve more problems will introduce more trade-offs too.

Re: Airflow's Problem

#90

Post by orwin »

MontyCarloHall wrote: post #7

Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor. Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is: My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture]…

Honestly, we have to set up airflow at my job for some datalog collection and treatment. Which is fine, only i'm pretty sure we had exactly the same issue at my old job that we fixed in half a day, including testing and deployment, with a perl script. And i think in this particular instance (gitlab logs) it was treated with 90% Awk. Meanwhile my coworkers still have issues after almost a week (not all of this is on airflow, but still).

I'm not saying Airflow is bad (we did set up a lot of hadoop clusters and other apache products at my old job, and our clients used airflow a lot), but i think the evangelists are so good they push airflow for everything, and this is bad. OP did use airflow for something it was not really designed for, and it sucked, but i do have this impression that tech writers and apache evangelists deserve some of the blame.

Post reply on HN