11–20 of 68 posts
There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so
Or people working to make some *nix tools work on different distributions.
Earlier quoted context omitted.
Curious what you like better?
not OP, but I prefer https://github.com/azkaban/azkaban to airflow
Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x). I realize you are buying not just the compute, but the management, but that e…
I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ against those outages.
I used to manage an airflow deployment (of which my team was the primary consumer), and it was not enjoyable in the least.
I wonder how it compares to astronomer.io, and Google's managed airflow thing.
disclosure: co-founder of astronomer
Earlier quoted context omitted.
not OP, but I prefer https://github.com/azkaban/azkaban to airflow
so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so
Alas, this relies on many things, not the least of which are peering agreements between cloud vendors that do not punish the consumer for using a service on one cloud with services from another.
It is possible to an extent today, but for competitive reasons, cloud companies do not seem to have a built in incentive to collaborate. Perhaps the cost of switching and lock-in will force cloud providers to work together in the long one, else they risk alienating customers, and then lack the ability to capture a larger market.
In other words, competition is good as long as people can realistically take advantage of it. I think cloud vendors have an obligation to see that this competition is encouraged and supported.
But why Airflow, it has so many weird things. I hope it is dethroned soon.
But why Airflow, it has so many weird things. I hope it is dethroned soon.
But why Airflow, it has so many weird things. I hope it is dethroned soon.
Earlier quoted context omitted.
so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.