Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

101–110 of 126 posts

Re: Airflow's Problem

#101

Post by robertlagrant »

ForHackernews wrote: post #68
jon_adler wrote: post #43

Is there any love for the Argo [1] project suite (Workflows, Events, CD) for this type of use case? I haven’t tried it out myself yet however it does look interesting. [1] https://argoproj.github.io

I've never used their workflows thing, but having been forced to live with ArgoCD it sounds horrifying. Argo is another over-engineered "CNCF" thing trying to ride the Kubernetes hype train. It's all "eventually consistent", which makes it extraordinarily difficult to see when any particular thing actually happened. Is my code deployed? Who knows, Argo is "syncing". Check out these great docs: https://argoproj.github…

How do you know if anything is deployed if it hasn't come back and confirmed it's deployed? Manual only?

Re: Airflow's Problem

#102

Post by carlsborg »

jdoliner wrote: post #85

I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's…

Cool. Are there any published benchmarks on how the data versioning engine scales?

Re: Airflow's Problem

#103

Post by correlator »

zibarn wrote: post #61
Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

Airflow with a celery backend is a pretty sweet combination. In that instance, airflow just gives you a nice scheduler to manage all the celery jobs.

Re: Airflow's Problem

#104

Post by JoshCole »

zibarn wrote: post #61
Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Not experienced here but as a genuine interest can you tell what problems airflow solves that can't be handled by celery and rabbitmq?

Expressing the problem you are trying to solve as a DAG is idiomatic in Airflow, but expressing your problem in terms of queue processing is idiomatic in Celery.

    a b c d vs. a (bc) d
They make different design decisions about what to surface via UX and what to make easy as a consequence of thinking of the problems in terms of different data structures.

Re: Airflow's Problem

#105

Post by kmbfjr »

dominotw wrote: post #26

Just put everything in a warehouse( via fivetran or some such thing) and just use DBT. Use airflow as cron runner for dbt. If you don't need realtime metrics, this formula works way better than convoluted airflow dags.

If you need cron, use cron or a cronjob pod. Airflow is a poor cron scheduler at any sort of scale.

Re: Airflow's Problem

#106

Post by jdoliner »

carlsborg wrote: post #102
jdoliner wrote: post #85

I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's…

Cool. Are there any published benchmarks on how the data versioning engine scales?

We are doing a whole bunch of performance testing on the new 2.0 engine that we released at the end of 2021. We'll be publishing those.

Re: Airflow's Problem

#107

Post by eru »

llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

Do you know what the author means by 'left' here? Probably not moving bits around in a way that equivalent to multiplying by powers of 2?

Re: Airflow's Problem

#108

Post by sbierwagen »

jdoliner wrote: post #106
carlsborg wrote: post #102

Earlier quoted context omitted.

Cool. Are there any published benchmarks on how the data versioning engine scales?

We are doing a whole bunch of performance testing on the new 2.0 engine that we released at the end of 2021. We'll be publishing those.

The end of 2021?

Re: Airflow's Problem

#109

Post by longcommonname »

Hippocrates wrote: post #47

I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (env…

Try gcsfuse to write logs to a bucket.

It's the one thing I like about our airflow. Everything else you said is echoed.

Also, the toil of dealing with many airflow instances when you have engineers who don't want to automate it.

Re: Airflow's Problem

#110

Post by dominotw »

kmbfjr wrote: post #105
dominotw wrote: post #26

Just put everything in a warehouse( via fivetran or some such thing) and just use DBT. Use airflow as cron runner for dbt. If you don't need realtime metrics, this formula works way better than convoluted airflow dags.

If you need cron, use cron or a cronjob pod. Airflow is a poor cron scheduler at any sort of scale.

yea for sure. Not sure what a good cron runner with web ui with logs is, jenkins?
Post reply on HN