31–40 of 50 posts
This looks pretty interesting. I am currently implementing DS workflows that are essentially python classes to orchestrate R scripts. I'll have a closer look on Monday but if I can use it to handle rpy2 R format data I'll be happy
Are you using reticulate ( https://github.com/rstudio/reticulate ), or having Python spawn a new worker process for R?
> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…
It would be much clearer if they compared side by side a few simple examples with regular makefiles that do the same thing, and people could see the advantages.
> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…
This library has a GPDR consent problem-- just putting it at the end of the README doesn't cut it. I wonder if this is the first open source library to violate GPDR.
Would be interested to find out what other tools are available in this space and how this compares with them. What’s the difference beteeen d6tflow, Luigi, Airflow?
I recently read that there are over 100 workflow engines [1], it is still often difficult to make the case that Make is insufficient [1] https://vatlab.github.io/blog/post/sos-workflow-engine/
It's easy to make the case that make is insufficient. There are three very distinct domains being conflated here: 1. The day to day experimentation and iteration by a computational researcher. 2. The repeated execution of a workflow on different data sets submitted by different people, such as in a clinical testing lab. 2. The ongoing processing of a stream of data by a deployed system, such as ongoing data processin…
I've never seen this work in practice, and doubt it can work, due to the complexities involved.
> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…
Earlier quoted context omitted.
This library has a GPDR consent problem-- just putting it at the end of the README doesn't cut it. I wonder if this is the first open source library to violate GPDR.
gdpr really only cares about personally identifiable information, afaik. You don't need consent for anonymized stats.
I feel so frustrated by the emergence of these things and the constant attempt to brand them or stylize them towards data science. There are just engineering projects. There are not any other things. For some engineering projects, you need to support dashboard-like, interactive interfaces that depend on data assets or other assets (like a database connection, a config file, a static representation of a statistical mo…
You're not wrong about the value of all the different tools you mention, but I think overlooking the integration and maintenance costs that a specialty tool can reduce, at the expense of some flexibility. I think that's the same reason many people prefer an IDE.