Flyte: A Cloud Native Machine Learning and Data Processing Platform
1–10 of 20 posts
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#2How does this differ from Airflow? https://airflow.apache.org/
Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#3How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.
Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences
- Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf
- Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface declaration
- Flyte is container and kibernetes native. It is also multi tenant.
- Flyte corn scheduler, control plane api and the actual execution engine are decoupled. Each workflow can be independently executed on a different execution engine
- Flyte workflows are purely specification - defined in protobuf and Flyte tasks also
- Flyte provides an event stream of the execution
- since Flyte is aware of the data, it comes with built in memorization and auto cataloging
- like airflow Flyte can have plugins in python, but it supports a richer plugin interface
- Flyte is written in Golang and on top of kuberenetes
It is definitely less mature in the open source, so please help us make it better. But it has been battle tested at Lyft for more than 3 years in production.
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#4Any plans to support Python type annotations, instead of using the @inputs and @outputs decorators?
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#5How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.
Here is a blog post I wrote a few weeks ago that describes how Flyte's structured workflow specification allows for using open-source workflows with no code.
https://medium.com/@flytehub/introducing-flytehub-open-sourc...
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#6Any plans to support Python type annotations, instead of using the @inputs and @outputs decorators?
That is something to which the SDK was designed to be extended--in fact, the SDK was designed specifically to be extended to different programming models besides the rudimentary one we provide out-of-box. However, we have not had the time to take on the implementation ourselves as the existing one serves our purposes at Lyft well and we are but a small team. So we do hope that someone in the community with a knack for syntax can help improve things!
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#7"use cached versions of pre-computed artifacts" This is really nice. I wonder if this also covers partial pre-computations, for example when the same subquery is reused across several pipelines.
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#8How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.
Airflow just cover authoring, monitoring and scheduling. I don't recall it covered caching, lineage nor resource allocation/reuse.
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#9Any plans to support Python type annotations, instead of using the @inputs and @outputs decorators?
You mean DataClasses?
Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform
#10Any plans to support Python type annotations, instead of using the @inputs and @outputs decorators?
You mean DataClasses?
No, I mean type annotations in the functions defining the tasks. If you look at their first example, they define a function called `get_traintest_splitdatabase`, and the input and outputs are annotated using two decorators.