Kedro: open-source library for production-ready machine learning code
1–10 of 20 posts
Re: Kedro: open-source library for production-ready machine learning code
#2I really like how they implemented the data catalog [0] so that it’s yaml-based and also has a paths-style cascading method of files that can be common across or within teams as well as personal for individual projects. I think this makes it easy to build up with tools for meta analysis (how many data sets are used, etc) and even viz using a variety of tools rather than having the metadata management tied to a system or product.
Are there other techniques for data catalogs that are file based or at least open standard based that scale all the way up from developer?
[0] https://kedro.readthedocs.io/en/latest/04_user_guide/04_data...
Re: Kedro: open-source library for production-ready machine learning code
#3Conjecture: production quality of ml code has mostly to do with how heuristics are designed and battle tested and almost nothing to do with how the training/inference pipeline is constructed.
Re: Kedro: open-source library for production-ready machine learning code
#4tldr, if you really dig past the marketing (from the FAQ (1)):
> We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers.
> Create the data transformation steps as pure Python functions
Personally, I feel mystified why you would use something like this rather than a more mature product like say, Spark, that natively supports clustering, etc, which is what I would really like to see in the FAQ.
Is it a processing solution? Not really, since it suggests you can offload the heavy lifting to an engine, eg. spark. An orchestrator? Apparently not, because that's a complementary product. So... it's like, a configuration management tool?
Pretty hard to see the use case to me.
1. https://kedro.readthedocs.io/en/latest/06_resources/01_faq.h...
Re: Kedro: open-source library for production-ready machine learning code
#5tldr, if you really dig past the marketing (from the FAQ (1)): > We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers. > Create the data transformation steps as pure Python functions Personally, I feel mystified why you would…
It makes a lot of sense to me, who struggled to work with a group consist of mostly Computer Vision scientists. The pushback to use anything heavier than pure python + s3 is amazing. Spark is still considered too heavy for us and people need to do in memory analysis and friction free experiments with convoluted dependencies. Most of the time it’s not their pipeline that goes to production but rather their trained model. They need to do most steps on their laptop and easy lift it to cloud. No engineers want to do that lift job trust me.
Netflix has Metaflow for this which seems a more fully-fledged product. But it’s not open sourced.
Re: Kedro: open-source library for production-ready machine learning code
#6tldr, if you really dig past the marketing (from the FAQ (1)): > We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers. > Create the data transformation steps as pure Python functions Personally, I feel mystified why you would…
Because running Spark to do anything that doesn’t actually require a whole cluster is like using earthmoving equipment to assemble a series of small ikea tables?
Re: Kedro: open-source library for production-ready machine learning code
#7> Machine learning models which can be deployed effortlessly and operate unattended are far more likely to achieve commercial objectives.
Likeliness of achieving commercial objectives is tied to the commercial usefulness and accuracy of your analysis and predictions, not the ease of deployment, or-even more curiously-ability to be left unattended.
Re: Kedro: open-source library for production-ready machine learning code
#8Starting to see a lot of these frameworks pop up to simplify deployment of machine learning models. I’m really hoping one or two start to stand out...but it doesn’t feel like this one.
Re: Kedro: open-source library for production-ready machine learning code
#9tldr, if you really dig past the marketing (from the FAQ (1)): > We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers. > Create the data transformation steps as pure Python functions Personally, I feel mystified why you would…
Because running Spark to do anything that doesn’t actually require a whole cluster is like using earthmoving equipment to assemble a series of small ikea tables?
If you're doing something that trivial, you don't need anything more complicated than airflow.
Re: Kedro: open-source library for production-ready machine learning code
#10I really like how they implemented the data catalog [0] so that it’s yaml-based and also has a paths-style cascading method of files that can be common across or within teams as well as personal for individual projects. I think this makes it easy to build up with tools for meta analysis (how many data sets are used, etc) and even viz using a variety of tools rather than having the metadata management tied to a system…
There's the intake project from the Anaconda folks.