Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Managed Workflows for Apache Airflow

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: AWS Managed Workflows for Apache Airflow

#41

Post by yandie »

rywalker wrote: post #37
rywalker wrote: post #14

Earlier quoted context omitted.

i'd let you know if their service came up :) been waiting an hour after hitting the "create" button disclosure: co-founder of astronomer

service never came up; also can't delete it :( "Environments with CREATING status must complete previous operation before initiating a new operation." can't email support for help (i only have basic plan) any AWSMWAA ppl on this thread and can help? the instance name is `airflow-ry-test`.

I think you'll need the account number for them to find out :P.

Re: AWS Managed Workflows for Apache Airflow

#42

Post by lmeyerov »

herodoturtle wrote: post #23
crispyporkbites wrote: post #3

There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version Is this a good idea? I don’t think so

> Is this a good idea? I don’t think so I agree. It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge…

they could submit their patches and extensions for OSS

but are largely not, narrowing to sharing core tweaks, and keeping the PaaS layers proprietary

so it's not the maintainers failing to pick & unify, but Google etc employees not giving back

Re: AWS Managed Workflows for Apache Airflow

#43

Post by seddonm1 »

vanpythonista wrote: post #35
seddonm1 wrote: post #24

Earlier quoted context omitted.

I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.

Agree that YAML is not really ideal for this case.

For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results

Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...

Re: AWS Managed Workflows for Apache Airflow

#44

Post by kumare3 »

vanpythonista wrote: post #35
seddonm1 wrote: post #24

Earlier quoted context omitted.

I am hoping Argo Workflows ( https://argoproj.github.io/projects/argo ) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.

Have you tried Flyte - flyte.org?

Re: AWS Managed Workflows for Apache Airflow

#45

Post by kumare3 »

dfsegoat wrote: post #8
mchusma wrote: post #6

Earlier quoted context omitted.

Curious what you like better?

Luigi from Spotify: https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.

Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

Re: AWS Managed Workflows for Apache Airflow

#46

Post by mchusma »

jozzy-james wrote: post #36
ttymck wrote: post #13

Earlier quoted context omitted.

Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)? I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ again…

not OP, nor do we use airflow (not sure there is a fit for us) - but $300-500 for something that could cost our modest company an equivalent half day of human time to mitigate, with possible tens of thousands in lost revenue if it occurs at the worst possible time, seem like a win-win peace of mind proposition.

Sure. The right circumstances this can make sense. But at a high level of your SAAS starts at $500/mo, you are selling to mid market and enterprises, not startups, almost universally. I was hoping for a product that would make sense for smaller companies. The list of business critical software we pay less for is long (GitHub, credit card processor, email, sms, managed redis, managed postgres, etc).

They can certainly sell this for what they want, I just won't buy at this price.

Re: AWS Managed Workflows for Apache Airflow

#47

Post by big_dinghy »

Want to shout out an alternative Python open source workflow orchestration tool, Prefect https://www.prefect.io/

It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.

Re: AWS Managed Workflows for Apache Airflow

#48

Post by jozzy-james »

mchusma wrote: post #46
jozzy-james wrote: post #36

Earlier quoted context omitted.

not OP, nor do we use airflow (not sure there is a fit for us) - but $300-500 for something that could cost our modest company an equivalent half day of human time to mitigate, with possible tens of thousands in lost revenue if it occurs at the worst possible time, seem like a win-win peace of mind proposition.

Sure. The right circumstances this can make sense. But at a high level of your SAAS starts at $500/mo, you are selling to mid market and enterprises, not startups, almost universally. I was hoping for a product that would make sense for smaller companies. The list of business critical software we pay less for is long (GitHub, credit card processor, email, sms, managed redis, managed postgres, etc). They can certainly…

i guess i really don't understand what separates a startup from enterprise...i know my particular company is nowhere near enterprise level...at least as far as i know of it, but i also know we're not a startup.

edit: in this sense...i know the fundamental differences otherwise

edit #2: we're also not a SaaS, but a B2C

Re: AWS Managed Workflows for Apache Airflow

#49

Post by jozzy-james »

jozzy-james wrote: post #48
mchusma wrote: post #46

Earlier quoted context omitted.

Sure. The right circumstances this can make sense. But at a high level of your SAAS starts at $500/mo, you are selling to mid market and enterprises, not startups, almost universally. I was hoping for a product that would make sense for smaller companies. The list of business critical software we pay less for is long (GitHub, credit card processor, email, sms, managed redis, managed postgres, etc). They can certainly…

i guess i really don't understand what separates a startup from enterprise...i know my particular company is nowhere near enterprise level...at least as far as i know of it, but i also know we're not a startup. edit: in this sense...i know the fundamental differences otherwise edit #2: we're also not a SaaS, but a B2C

and i totally get where you're coming from - i'm the point guy on validating any new integrations for what we do. i was just commenting on the cost/benefit of having someone fix a home rolled version vs their managed and if it'd save money
Post reply on HN