Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

61–70 of 126 posts

Re: Airflow's Problem

#61

Post by zibarn »

Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

Re: Airflow's Problem

#62

Post by orthoxerox »

A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure.

Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS generated, but the ops don't care either way, since Airflow doesn't care what it's running.

Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem.

"What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem.

"Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem.

Re: Airflow's Problem

#63

Post by geertj »

peteradio wrote: post #35
geertj wrote: post #2

We tried to set up Airflow in our team in the past. The big problem we encounrted is that its unit of management (I believe it's called a "job" but I'm rusty on this) is too low level. Our pipeline processes a lot of data and we have millions of jobs per day. Once Airflow has an (planned or unplanned) outage, 10s of thousands of job start piling up, and it never recovers from that. In the end we replaced our data orc…

What do you do for repeated failures? Does it get flagged for a manual debug or does it kick into a different mode of automation?

We notice repeated failures because we have metrics on our "up to dateness", and those metrics will stall. We also send logs to CloudWatch logs and alarm on certain threshold of errors. Once an alarm fires, we investigate manually and see why the job is failing. This happens occasionally but not too much. While we are investigating, we are spinning up repeat jobs with some frequency, but this hasn't proved to be a problem.

Re: Airflow's Problem

#64

Post by abirch »

llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

I've seen data errors because of joins and aggregations. Data democratization can be a net negative, especially if people don't question the graphs they see.

Re: Airflow's Problem

#65

Post by shmoogy »

glogla wrote: post #16
shmoogy wrote: post #15

I've been using airflow for about 2 years now in production. It's been mostly good - the few times things go wrong, it's a huge pain in the ass to figure out why... but it's significantly better than just straight cron on Linux. Airflow 2 has improved a lot of speed and catching up issues from airflow 1.x I don't have time to investigate other solutions like dagster and prefect and migrate jobs to it for testing.

If it is just you, you are fine, and I'm not sure other tools would have that much benefit. Trouble with Airflow starts when multiple teams and user types start to share it.

I've definitely noticed more issues after adding users, but it's more that they don't actually understand a lot of what they're trying to do and cause problems when writing dags.

Re: Airflow's Problem

#67

Post by abirch »

mumblemumble wrote: post #44
llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt. It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a who…

I'd say that Big Data is the Collateralized Debt Obligations of business operations. It looks fabulous from afar but it can blow things up quickly if there's no understanding of the internals.

Re: Airflow's Problem

#68

Post by ForHackernews »

jon_adler wrote: post #43

Is there any love for the Argo [1] project suite (Workflows, Events, CD) for this type of use case? I haven’t tried it out myself yet however it does look interesting. [1] https://argoproj.github.io

I've never used their workflows thing, but having been forced to live with ArgoCD it sounds horrifying.

Argo is another over-engineered "CNCF" thing trying to ride the Kubernetes hype train. It's all "eventually consistent", which makes it extraordinarily difficult to see when any particular thing actually happened. Is my code deployed? Who knows, Argo is "syncing".

Check out these great docs: https://argoproj.github.io/argo-workflows/rest-api/

> API reference docs :

> Latest docs (maybe incorrect)

> Interactively in the Argo Server UI.https://localhost:2746/apidocs> (>= v2.10)

Yes, that is a localhost URL on their website.

Re: Airflow's Problem

#69

Post by Hippocrates »

zibarn wrote: post #61
Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

I have not used celery + rabbitmq but I assume that combo is like sidekiq + redis, or any other job queue + worker system.

Airflow packages those things together and adds some additional features - UI with Graph, gantt, logs and other views of the workflow - Users and permissions - Places to store config - Mechanisms for passing small data between tasks - Various "sensors" for triggering workflows - Various operators that interact with common data-oriented systems (bigquery, snowflake, s3, you name it). These are basically libraries that expose a config-forward API.

Probably the main selling point is the pre-made operators, but in short it is a complete solution with bells and whistles that aligns itself with the data ecosystem.

Re: Airflow's Problem

#70

Post by morelisp »

lopatin wrote: post #53
dusted wrote: post #27

Earlier quoted context omitted.

I'd be okay with the article headline on HN being "Apache Airflow's Problem" so that I know it's about a piece of apache software and not something interesting about airflow.

I think context matters and the title "Airflow's Problem" doesn't make much sense when talking about the physical phenomenon of flowing air.

Not really, the past two years have seen probably a order of magnitude increase in interest and articles about industrial-scale ventilation and air quality.
Post reply on HN