Viewing profile — kumare3

[dead]

May I recommend looking into flyte.org, it is open source kubernetes native "orchestration" style tool, but essentially and infrastructure component that is geared to making your M…

Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte

I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows yo…

Checkout Flyte.org and it’s sibling project https://www.union.ai/unionml

@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available Local…

Some of these were the core problems that we wanted to address as part of https://flyte.org . We started with a team first and multi-tenant approach at the core. For example, each …

I remember having this feeling a few years ago. What I realized is that airflow has taught us a few bad habits and also brought ahead an interesting paradigm of the vertical workfl…

This is cool, you could provide resilience to Red Engine, by providing it a backend using Flyte.org. Checkout example of making a new API on top of Flyte which has similarish feel …

+1

Thank you. So at Lyft, there exists an AirflowFlyteOperator. It was designed to interop with Airflow. I know the community is working on open sourcing it. Keep a lookout for the sa…

Ohh you are welcome, join the slack channel and ask for help. The community is growing everyday - here are some examples of using it in python https://flytecookbook.readthedocs.io/…

Have you tried Flyte.org?

Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

Have you tried Flyte - flyte.org?

Roberto, this is absolutely one of our goals. When we started, it was with python2.7 still around. We would love contributions, ofcourse we will work with you and adapt it

Another great question. So Airflow is used at Lyft for ETL. I think for traditional ETL it still is a good fit. But, there is an effort to not just migrate, but rethink how we can …

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important diff…