D6tflow: Python library for building data science workflows
1–10 of 50 posts
Re: D6tflow: Python library for building data science workflows
#2- I really don't think that Python needs an additional layer of abstraction, especially when using scikit-learn which is high level enough as it is.
- The workflow in the readme is missing the part when you actually use the model. This will often need to be connected to your original preprocessing in some way - for example, if the dataset that you're predicting on has a categorical variable with a unique value which wasn't present in the training dataset, this effectively introduces a new feature in your dataset, which makes it impossible to do model.predict(). The need to manage things like this changes that workflow chart quite a bit.
Re: D6tflow: Python library for building data science workflows
#3This looks pretty interesting. I am currently implementing DS workflows that are essentially python classes to orchestrate R scripts. I'll have a closer look on Monday but if I can use it to handle rpy2 R format data I'll be happy
Re: D6tflow: Python library for building data science workflows
#4"For data scientists and data engineers, d6tflow is a python library which makes building complex data science workflows easy, fast and intuitive. It is built on top of workflow manager luigi but unlike luigi it is optimized for data science workflows."
But they didn't really explain/sell what those optimisations are in the readme.
Re: D6tflow: Python library for building data science workflows
#5This looks pretty interesting. I am currently implementing DS workflows that are essentially python classes to orchestrate R scripts. I'll have a closer look on Monday but if I can use it to handle rpy2 R format data I'll be happy
I've been wondering about this - why would you pass data from one language to the other? If you already have existing R code, you can easily manage it through R itself.
Re: D6tflow: Python library for building data science workflows
#6This looks pretty interesting. I am currently implementing DS workflows that are essentially python classes to orchestrate R scripts. I'll have a closer look on Monday but if I can use it to handle rpy2 R format data I'll be happy
I've been wondering about this - why would you pass data from one language to the other? If you already have existing R code, you can easily manage it through R itself.
I suppose we could have written an R API to serve ML results directly, but there are a lot of other auxiliary tasks needed like protobuf handling, AWS integration and some other business logic that means python makes more sense. The quickest way to make it work was to wrap the ML scripts (and they really are __scripts__, not even functions..) in python and just handle everything outside of R. I think eventually we will just re-implement the ML stuff in python anyway.
Re: D6tflow: Python library for building data science workflows
#7For another way to do data science workflows, there's the Common Workflow Language (http://commonwl.org)
Re: D6tflow: Python library for building data science workflows
#8- I really don't think that Python needs an additional layer of abstraction, especially when using scikit-learn which is high level enough as it is. - The workflow in the readme is missing the part when you actually use the model. This will often need to be connected to your original preprocessing in some way - for example, if the dataset that you're predicting on has a categorical variable with a unique value which…
The full machine learning pipeline goes far beyond sklearn scope. Think about scheduled data reading or scheduled model updates. I think there is a need for such frameworks to help using ml.
Re: D6tflow: Python library for building data science workflows
#9- I really don't think that Python needs an additional layer of abstraction, especially when using scikit-learn which is high level enough as it is. - The workflow in the readme is missing the part when you actually use the model. This will often need to be connected to your original preprocessing in some way - for example, if the dataset that you're predicting on has a categorical variable with a unique value which…
The full machine learning pipeline goes far beyond sklearn scope. Think about scheduled data reading or scheduled model updates. I think there is a need for such frameworks to help using ml.
Going far beyond the scope of sklearn pipelines isn't necessarily a good thing. I don't really see what there is to be gained by making scheduling part of the remit of pipelines.
I'd rather assemble together a set of tightly scoped "UNIX philosophy" libraries and tools rather than try and use an all encompassing framework and be straitjacketed by its imposed structure.
At work I have what are essentially cron jobs running scripts which invoke sklearn pipelines. I've never even thought to make the scheduler aware of what they were running and I'm not sure why I would.
Re: D6tflow: Python library for building data science workflows
#10For another way to do data science workflows, there's the Common Workflow Language ( http://commonwl.org )
Most of what I've heard about CWL is that it's unwieldy to use directly; it's main value is as a backend or as an interchange format between different workflow systems. However, I haven't tried it myself and would be interested to hear others' experiences on it.