But why Airflow, it has so many weird things. I hope it is dethroned soon.
Curious what you like better?
temporal.io
AWS Managed Workflows for Apache Airflow
31–40 of 68 posts
Re: AWS Managed Workflows for Apache Airflow
#32I really wish managed airflow instances were cheaper for smaller companies. I built my own using spot instances and it's so affordable compared to astronomer and the others.
So far it's been very low maintenance - outside of the few random scares where dag logs filled up my server - but then I researched and found maintenance Dags that prevent a lot of issues.
Wondering when my next outage will be is always fun, but it's been pretty stable so far.
E: I know and appreciate the tech they put into it. It's just too high of a price for me once I get the workers added. I still want migrate mine to fargate workers at some point though.
Re: AWS Managed Workflows for Apache Airflow
#33Thank you AWS. I think they just saved me $85K.
Re: AWS Managed Workflows for Apache Airflow
#34But why Airflow, it has so many weird things. I hope it is dethroned soon.
Lmao this is a great comment - I have found so many bizarre eccentricities with Airflow, but it is still, in general, quite nice to build with. What are you thinking about?
Re: AWS Managed Workflows for Apache Airflow
#35But why Airflow, it has so many weird things. I hope it is dethroned soon.
I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
Re: AWS Managed Workflows for Apache Airflow
#36I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x). I realize you are buying not just the compute, but the management, but that e…
Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)? I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ again…
not OP, nor do we use airflow (not sure there is a fit for us) - but $300-500 for something that could cost our modest company an equivalent half day of human time to mitigate, with possible tens of thousands in lost revenue if it occurs at the worst possible time, seem like a win-win peace of mind proposition.
Re: AWS Managed Workflows for Apache Airflow
#37I wonder how it compares to astronomer.io, and Google's managed airflow thing.
i'd let you know if their service came up :) been waiting an hour after hitting the "create" button disclosure: co-founder of astronomer
service never came up; also can't delete it :(
"Environments with CREATING status must complete previous operation before initiating a new operation."
can't email support for help (i only have basic plan)
any AWSMWAA ppl on this thread and can help? the instance name is `airflow-ry-test`.
Re: AWS Managed Workflows for Apache Airflow
#38Thank you AWS. I think they just saved me $85K.
firing your devops person? maybe wait a few weeks to be sure!
Re: AWS Managed Workflows for Apache Airflow
#39There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so
I haven't tried it yet but this honestly looks pretty close to vanilla, the post even says that they contributed their patches to upstream. What differences have you spotted?
Of course their storage backend is going to be S3 and they are going to send logs to Cloudwatch, we have been doing it in a similar way for quite some time and it is what I expect from a solution managed by AWS.
Re: AWS Managed Workflows for Apache Airflow
#40There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so
I respectfully disagree. Not only do cloud native versions of an OSS encourages innovation and enrichment, it also lowers the operational cost of running it in-house and aligns closely to Devops principles.