Using Argo to Train Predictive Models at FlightAware
flightaware.engineering
Using Argo to Train Predictive Models at FlightAware
1–10 of 14 posts
Re: Using Argo to Train Predictive Models at FlightAware
#2Out of curiosity, was Kubeflow ever considered and was there a deliberate decision against Kubeflow?
Kubeflow Pipelines under the hood (and not very hidden) uses Argo Workflows.
Re: Using Argo to Train Predictive Models at FlightAware
#3Really like the flexibility provided by Argo. It's the missing workflow concept from Kubernetes.
Re: Using Argo to Train Predictive Models at FlightAware
#4Out of curiosity, was Kubeflow ever considered and was there a deliberate decision against Kubeflow? Kubeflow Pipelines under the hood (and not very hidden) uses Argo Workflows.
Or Apache Airflow. I'd expect this and KFP to have better integrations that cater to data science. You can use Elyra extension to design the workflows right from JupyterLab.
Re: Using Argo to Train Predictive Models at FlightAware
#5Out of curiosity, was Kubeflow ever considered and was there a deliberate decision against Kubeflow? Kubeflow Pipelines under the hood (and not very hidden) uses Argo Workflows.
Or Apache Airflow. I'd expect this and KFP to have better integrations that cater to data science. You can use Elyra extension to design the workflows right from JupyterLab.
Not from said company, but did do a workflow tool comparison at my work and we also went with Argo.
Argo had better kubernetes + surrounding ecosystem integration out of the box, it was designed to run containers by default which suited us because we had mixed language workloads. Airflow was mostly Python specific, unless you then ran plugins and extensions, the config/pipeline definition was written in Python which I didn’t want to do after witnessing my teammates write the worst Python I’ve seen in my career, and last time I evaluated it, it depended on a bunch of external, Python specific tools (celery etc) that I had previously found painful to run.
Re: Using Argo to Train Predictive Models at FlightAware
#6Argo is extremely capable at its focus — coordinating workflows on Kubernetes.
I had used Airflow for a few years, and looked into Prefect; in retrospect I'm very happy we chose Argo.
Use Argo if: - Your tasks are containerized. - You're using Kubernetes, and can benefit from what it can offer — individually sized containers, autoscaling, fault-tolerance. - You have loosely coupled tasks — which pass at most pass files to each other, rather than python objects. - You don't have tens of thousands of tasks / streaming / etc.
Airflow can run on Kubernetes, but with Airflow we ended up having equally sized workers up 24/7 — whether or not it was running an expensive job, a query on a remote system, or nothing.
Re: Using Argo to Train Predictive Models at FlightAware
#7Out of curiosity, was Kubeflow ever considered and was there a deliberate decision against Kubeflow? Kubeflow Pipelines under the hood (and not very hidden) uses Argo Workflows.
I found Kubeflow Pipelines to be worse than vanilla Argo. Not only is the documentation poor for the python DSL that gets compiled to the Argo workflow spec but some operations are downright impossible to express in the DSL. Vanilla Argo + RBAC works wonderfully especially with Argo Server since v3+.
Re: Using Argo to Train Predictive Models at FlightAware
#8Why create a new model for each airport? That doesn't seem right. Delays are compounding and many are regional.
Re: Using Argo to Train Predictive Models at FlightAware
#9This is very cool! If you are interested in using a similar setup for your ML workloads (Argo + K8S), I'd love to chat with you (ville@outerbounds.co).
We are working on integrating Argo & K8S for Metaflow, an open-source ML framework originally developed at Netflix https://github.com/Netflix/metaflow/pull/434
Re: Using Argo to Train Predictive Models at FlightAware
#10Out of curiosity, was Kubeflow ever considered and was there a deliberate decision against Kubeflow? Kubeflow Pipelines under the hood (and not very hidden) uses Argo Workflows.
I found Kubeflow Pipelines to be worse than vanilla Argo. Not only is the documentation poor for the python DSL that gets compiled to the Argo workflow spec but some operations are downright impossible to express in the DSL. Vanilla Argo + RBAC works wonderfully especially with Argo Server since v3+.
I agree. For anything sophisticated you have to use the Kubernetes Python client combined with the KFP DSL. The compiled Argo Workflow spec can become such a mess that you quickly run into a size limitations -- especially if you choose to deploy a Python function directly (which then uses Kaniko to build the containers in cluster and run them via Argo Workflows). At that point I would rather use Argo directly. And KFP is generally quite behind in Argo Workflow versions.
And it's not like KFP makes CI/CD (MLOps) any easier than Argo Workflows itself.