Confluent seems to be thriving ever since AWS launched their own Kafka managed service. With that as an example, this could be a good thing for Astronomer.io.
AWS Managed Workflows for Apache Airflow
51–60 of 68 posts
Re: AWS Managed Workflows for Apache Airflow
#52But why Airflow, it has so many weird things. I hope it is dethroned soon.
fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...
It'd be really helpful if there were an article that folks could read.
Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video.
This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
Re: AWS Managed Workflows for Apache Airflow
#53But why Airflow, it has so many weird things. I hope it is dethroned soon.
fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...
Do you have an article or something I can skim/scan more quickly? Not going to invest over an hour to watch those videos.
Re: AWS Managed Workflows for Apache Airflow
#54How does this differ from Google's Cloud Composer, which is also managed Airflow? https://cloud.google.com/composer/
From the pricing examples AWS seems more expensive at least.
Re: AWS Managed Workflows for Apache Airflow
#55Earlier quoted context omitted.
fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...
It'd be really helpful if there were an article that folks could read. Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video. This trend towards posting video content instead of, rather than along-side, written content is p…
Here's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/
Re: AWS Managed Workflows for Apache Airflow
#56There's no mention of AWS's own existing workflow management systems - Step Functions and AWS Glue. Would be interested to see AWS's own advice on when to use one over the other.
Re: AWS Managed Workflows for Apache Airflow
#57There's no mention of AWS's own existing workflow management systems - Step Functions and AWS Glue. Would be interested to see AWS's own advice on when to use one over the other.
This was developed internally by the Orchestration organization - that builds Step Functions and maintains AWS Simple Workflow [1], or have you forgotten the original AWS workflow system :)
I don't think of Glue as a generic workflow system the way the others are - it's definitely much more optimized for ETL use cases.
With time, I'm sure there'll be more detailed guidance on Step Functions vs Apache Airflow, but the simple guidance might be that Step Functions is a fully AWS-native (and serverless) orchestration engine. Whereas, of course Apache Airflow is an open source project with a diverse ecosystem of other plugins.
Re: AWS Managed Workflows for Apache Airflow
#58Thank you AWS. I think they just saved me $85K.
how does that compute?
Re: AWS Managed Workflows for Apache Airflow
#59But why Airflow, it has so many weird things. I hope it is dethroned soon.
I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.
Re: AWS Managed Workflows for Apache Airflow
#60Earlier quoted context omitted.
Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow. I would be also curious to hear more about alternative and mature open-source solutions to Airflow. [1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably. Disclosure: I'm on the Airflow PMC.
Is there a list of backward incompatible changes?
Also, how would you describe the overall health of the code base with all of these new features added?