AWS Managed Workflows for Apache Airflow • Hacker News Board

Re: AWS Managed Workflows for Apache Airflow

#61

Post by ttsda » Wed, Nov 25, 2020, 10:10 AM UTC

mfateev wrote: post #31

mchusma wrote: post #6

Earlier quoted context omitted.

Curious what you like better?

temporal.io

This is it, the DAG model is a thing of the past

Re: AWS Managed Workflows for Apache Airflow

#62

Post by kfk » Wed, Nov 25, 2020, 11:09 AM UTC

I continue to be confused about AWS offerings other than their EC2 and data storage (S3/Redshift/Spectrum/Aurora) solutions which are undoubtedly amazing products. The thing with Workflow scheduling and orchestration is that it's complex and non trivial. I'd rather buy a product built by a company with a focus in this area (i.e. Prefect) rather than go with yet another poorly executed but highly marketed AWS product. When I buy a product for my team I buy support and integration first, technology is less important. A lot of AWS products are poorly supported and mostly integrate with other AWS stuff. On top of all of this we are supposed to be running containers and AWS makes all this serverless stuff which defeats the whole purpose of moving applications to containers smoothly running in the cloud.

Re: AWS Managed Workflows for Apache Airflow

#63

Post by polskibus » Wed, Nov 25, 2020, 12:37 PM UTC

mchusma wrote: post #5

I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x). I realize you are buying not just the compute, but the management, but that e…

What is there to manage in airflow that can be outsourced?

Re: AWS Managed Workflows for Apache Airflow

#64

Post by whoevercares » Wed, Nov 25, 2020, 12:46 PM UTC

howlgarnish wrote: post #50

How does this differ from Google's Cloud Composer, which is also managed Airflow? https://cloud.google.com/composer/

Most of the time this doesn’t have to be different, the conversation is likely a big enterprise customer says “do you have managed airflow? If not we might consider move(a little/some/all) of our workload to Google Cloud b/c they have it”

AWS: “we will build one in 3 months”

See. nobody care about the differentiator in big 2B enterprise business . It’s more about trust and migration cost

Re: AWS Managed Workflows for Apache Airflow

#65

Post by dataminded » Wed, Nov 25, 2020, 3:03 PM UTC

swyx wrote: post #58

dataminded wrote: post #33

Thank you AWS. I think they just saved me $85K.

how does that compute?

Our security posture requires that I purchase support for all software that we run in production. There are very few organizations offering support for Airflow and they aren't cheap.

Re: AWS Managed Workflows for Apache Airflow

#66

Post by dataminded » Wed, Nov 25, 2020, 3:04 PM UTC

rywalker wrote: post #38

dataminded wrote: post #33

Thank you AWS. I think they just saved me $85K.

firing your devops person? maybe wait a few weeks to be sure!

Nobody is getting fired (at our org at least). We have to purchase support for Airflow due to security policy. AWS is much cheaper than any alternatives.

Re: AWS Managed Workflows for Apache Airflow

#67

Post by swyx » Wed, Nov 25, 2020, 6:59 PM UTC

dataminded wrote: post #65

swyx wrote: post #58

Earlier quoted context omitted.

how does that compute?

Our security posture requires that I purchase support for all software that we run in production. There are very few organizations offering support for Airflow and they aren't cheap.

fair enough, thank you for taking the time to explain.

Re: AWS Managed Workflows for Apache Airflow

#68

Post by seddonm1 » Wed, Nov 25, 2020, 11:05 PM UTC

lxglv wrote: post #59

seddonm1 wrote: post #24

Earlier quoted context omitted.

I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.

Maybe.

In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.