Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Managed Workflows for Apache Airflow

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: AWS Managed Workflows for Apache Airflow

#21

Post by spfink »

crispyporkbites wrote: post #3

There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so

What is your ideal solution in these cases? In my experience, large enterprise customers (among others) tend to prefer managed versions of software where available. Cloud agnostic/interop versions would certainly seem like a good idea but I wonder what effect that would have on pricing and growth of the product.

Re: AWS Managed Workflows for Apache Airflow

#22

Post by dimberman »

six2seven wrote: post #9
mchusma wrote: post #6

Earlier quoted context omitted.

Curious what you like better?

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow. I would be also curious to hear more about alternative and mature open-source solutions to Airflow. [1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably.

Disclosure: I'm on the Airflow PMC.

Re: AWS Managed Workflows for Apache Airflow

#23

Post by herodoturtle »

crispyporkbites wrote: post #3

There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so

> Is this a good idea? I don’t think so

I agree.

It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge it, and use it - to the exclusion of all other implementations - not because of bias, but because of the truth underlying our meritocracy.

The question for me is - when we engage in said meritocracy, with all these BDFLs calling the shots, will they be the ones to negotiate with Jeff, Bill, and Larry? Or will we?

Re: AWS Managed Workflows for Apache Airflow

#24

Post by seddonm1 »

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

Re: AWS Managed Workflows for Apache Airflow

#25

Post by blakeburch »

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with.

I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.

Re: AWS Managed Workflows for Apache Airflow

#26

Post by GordonS »

ttymck wrote: post #13
mchusma wrote: post #5

I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x). I realize you are buying not just the compute, but the management, but that e…

Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)? I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ again…

(I'm not the OP)

For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely.

But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.

Re: AWS Managed Workflows for Apache Airflow

#28

Post by drewda »

seddonm1 wrote: post #24
brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-wor...

Re: AWS Managed Workflows for Apache Airflow

#29

Post by mchusma »

GordonS wrote: post #26
ttymck wrote: post #13

Earlier quoted context omitted.

Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)? I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ again…

(I'm not the OP) For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely. But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.

(am OP) We haven't had a major outage in 3 years of using airflow. Some issues, and it consumes some time, but so would managed airflow in all likelihood. Most issues are related to "airflow configuration type things" that presumably this solution would not fix.

I am really just surprised it is not a "no brainier price". It's a tough sell. I am sure it's valuable for some people, I just can't really justify using it. I think value to me is like $100/mo (as random airflow user on the internet).

Re: AWS Managed Workflows for Apache Airflow

#30

Post by mchusma »

dimberman wrote: post #22
six2seven wrote: post #9

Earlier quoted context omitted.

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow. I would be also curious to hear more about alternative and mature open-source solutions to Airflow. [1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably. Disclosure: I'm on the Airflow PMC.

Thanks for your work on airflow. 2.0 is exciting and we are looking forward to it. Love airflow 1 though too, it has helped us a ton.
Post reply on HN