Live data from Hacker News

Flyte: A Cloud Native Machine Learning and Data Processing Platform

eng.lyft.com

11–20 of 20 posts

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#11
post #7

"use cached versions of pre-computed artifacts" This is really nice. I wonder if this also covers partial pre-computations, for example when the same subquery is reused across several pipelines.

It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks which are individually tractable for testing and validation--especially as the tasks and pipelines evolve in a large organization! Further, the verbosity of authoring such pipelines can be reduced significantly by making good use of the flytekit library.

Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#12
post #3

How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf - Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface…

Have Lyft migrated all their workflows from Airflow to Flyte? Or does Airflow still play a role alongside Flyte? Was assuming Lyft is running workflows in Airflow from this post https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8f...

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#13
post #7

"use cached versions of pre-computed artifacts" This is really nice. I wonder if this also covers partial pre-computations, for example when the same subquery is reused across several pipelines.

It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks whic…

I see, thanks. I was hoping for this to be more automatic.

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#14
post #9
post #4

Any plans to support Python type annotations, instead of using the @inputs and @outputs decorators?

You mean DataClasses?

typing — Support for type hints [1]. They are used by tools like mypy, not at runtime (by CPython at least), but still very useful.

[1]: https://docs.python.org/3/library/typing.html

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#15

How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well.

One example I see is that one can run Airflow simply without containers if desired with just simple Python functions whereas Flyte seems to be much more concerned with managing the execution environment for you (pros/cons to that).

Flyte also seems to be more "Kubernetes native" by default [2][3] vs with Airflow this is more of a choice amongst several executors.

I'd be curious to see a performance benchmark using comparable workflows vs Airflow with the Kubernetes Executor or Kubernetes Operator.

[1]: https://www.prefect.io/

[2]: https://kccncna19.sched.com/event/UaYY/flyte-cloud-native-ma...

[3]: https://www.youtube.com/watch?v=KdUJGSP1h9U

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#16

How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well. One example I see is that one can run Airflow simply without containers if des…

Thanks for the link! Looking at Prefect's comparison to Airflow (https://docs.prefect.io/core/welcome/why_not_airflow.html#wh...), I think it sums up a lot of the same areas we tried to address with Flyte! Particularly the bullet-points in the 'Overview' section were all things, to the word, that were paramount for us to solve at Lyft. Now perhaps we need to publish an article on comparison to Prefect ;)

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#17

Earlier quoted context omitted.

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well. One example I see is that one can run Airflow simply without containers if des…

Thanks for the link! Looking at Prefect's comparison to Airflow ( https://docs.prefect.io/core/welcome/why_not_airflow.html#wh... ), I think it sums up a lot of the same areas we tried to address with Flyte! Particularly the bullet-points in the 'Overview' section were all things, to the word, that were paramount for us to solve at Lyft. Now perhaps we need to publish an article on comparison to Prefect ;)

Thank you for replying. I would definitely look forward to that post.

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#18
post #3

How does this differ from Airflow? https://airflow.apache.org/ Looking at the docs for both, Flyte has similar functionality to Airflow, except less mature and a more functional Task specification syntax. Airflow has the same data ETL operators as well, plus a few more.

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf - Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface…

Quite interesting to hear this, it's very much the same observations while working with some customers, Airflow is a very mature and an amazing tool, but it does not have a good state/artifacts management, which leaves the users tweaking around, scheduling is centralised, and is not designed for ML workflows, i.e. hyperparams tuning, distributed runs, ... the kubernetes support is also quite limited.

Polyaxon[0] took a similar approach to FLyte, i.e. for authoring specifications: strongly typed system in protobuf + intuitive yaml specification + sdks in Python/golang/java/... It also treats operations (tasks in Flyte) as first class citizens and allows to run them in a serverless way. Users can choose to register repetitive operations as components and share them with a description and a typed inputs/outputs.

[0] https://github.com/polyaxon/polyaxon

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#19
post #12
post #3

Earlier quoted context omitted.

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf - Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface…

Have Lyft migrated all their workflows from Airflow to Flyte? Or does Airflow still play a role alongside Flyte? Was assuming Lyft is running workflows in Airflow from this post https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8f...

Another great question. So Airflow is used at Lyft for ETL. I think for traditional ETL it still is a good fit. But, there is an effort to not just migrate, but rethink how we can leverage Flyte's capabilities to improve our ETL experience.

But, as it exists, we have a FlyteAirflowOperator, so that users can easily connect their Airflow pipelines with Flyte and write the new ones on Flyte alone.

Stay tuned for developments on this front :)

Re: Flyte: A Cloud Native Machine Learning and Data Processing Platform

#20
post #10
post #9

Earlier quoted context omitted.

You mean DataClasses?

No, I mean type annotations in the functions defining the tasks. If you look at their first example, they define a function called `get_traintest_splitdatabase`, and the input and outputs are annotated using two decorators.

Roberto, this is absolutely one of our goals. When we started, it was with python2.7 still around. We would love contributions, ofcourse we will work with you and adapt it
Post reply on HN