AWS Managed Workflows for Apache Airflow

#1

Post by soamv » Tue, Nov 24, 2020, 8:36 PM UTC

AWS Managed Workflows for Apache Airflow
aws.amazon.com

Re: AWS Managed Workflows for Apache Airflow

#2

Post by soamv » Tue, Nov 24, 2020, 8:39 PM UTC

I wonder how it compares to astronomer.io, and Google's managed airflow thing.

Re: AWS Managed Workflows for Apache Airflow

#3

Post by crispyporkbites » Tue, Nov 24, 2020, 9:36 PM UTC

There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version

Is this a good idea? I don’t think so

Re: AWS Managed Workflows for Apache Airflow

#4

Post by brainzap » Tue, Nov 24, 2020, 9:36 PM UTC

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Re: AWS Managed Workflows for Apache Airflow

#5

Post by mchusma » Tue, Nov 24, 2020, 9:41 PM UTC

I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x).

I realize you are buying not just the compute, but the management, but that ends up being something in the cost range of $300-$500 or so per month for the airflow management part of it. Seems a bit steep. $50-$100/mo would be a no brainer for us. For some orgs I can see this being a great solution, but its not really friendly for the little guy (with a min price of $350/mo).

Re: AWS Managed Workflows for Apache Airflow

#6

Post by mchusma » Tue, Nov 24, 2020, 9:42 PM UTC

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Curious what you like better?

Re: AWS Managed Workflows for Apache Airflow

#7

Post by kingkonz1 » Tue, Nov 24, 2020, 9:46 PM UTC

mchusma wrote: post #6

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Curious what you like better?

not OP, but I prefer https://github.com/azkaban/azkaban to airflow

Re: AWS Managed Workflows for Apache Airflow

#8

Post by dfsegoat » Tue, Nov 24, 2020, 9:53 PM UTC

mchusma wrote: post #6

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Curious what you like better?

Luigi from Spotify:

https://github.com/spotify/luigi

We’ve been using it for complex update workflows for about 5 yrs now, and it just works.

It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.

Re: AWS Managed Workflows for Apache Airflow

#9

Post by six2seven » Tue, Nov 24, 2020, 9:56 PM UTC

mchusma wrote: post #6

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Curious what you like better?

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow.

I would be also curious to hear more about alternative and mature open-source solutions to Airflow.

[1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

Re: AWS Managed Workflows for Apache Airflow

#10

Post by x87678r » Tue, Nov 24, 2020, 9:57 PM UTC

mchusma wrote: post #6

brainzap wrote: post #4

But why Airflow, it has so many weird things. I hope it is dethroned soon.

Curious what you like better?

https://dagster.io/