D6tflow: Python library for building data science workflows

I have a strong, negative reaction to this. I read the collection code (in d6collect), and it does as they claim (with perhaps minor qualms about what anonymous really means). And (for now) it's easy to disable without mucking around in the code. But in fact I'm not sure what I was really looking for, since I don't imagine using this library when I might need to reverify that they still aren't collecting anything I don't want to be collected.

On the other hand, I'm glad that they mentioned it --- I would have a much more negative reaction if I had to find this out on my own.

I am a long-time practitioner of putting machine learning tools into production, improving ML models over time, doing maintenance on deployed ML models, and researching new ways to solve problems with ML models.

All I can say is that in based on my experience, I would dramatically disagree with what you wrote.

I’ve always found pre-existing generalist engineering tooling to work more efficiently and cover all the features I need in a more reliable and comprehensive way than any of the latest and greatest ML-specific workflow tools of the past ~10 years.

I’ve also worked on many production systems that do not involve any aspects of statistical modeling, yet still rely on large data sets or data assets, offline jobs that perform data transformations and preprocessing, require extensibly configurable parameters, etc. etc.

I’ve never encountered or heard of any ML system that is in any way different in kind than most other general types of production engineering systems.

But I have seen plenty of ML projects that get bogged down with enormous tech debt stemming from adopting some type of fool’s gold ML-specific deployment / pipeline / data access tools and running into problems that time-honored general system tools would have solved out of the box, and then needing to hack your own layers of extra tooling on top of the ML-specific stuff.

It's easy to make the case that make is insufficient. There are three very distinct domains being conflated here:

1. The day to day experimentation and iteration by a computational researcher. 2. The repeated execution of a workflow on different data sets submitted by different people, such as in a clinical testing lab. 2. The ongoing processing of a stream of data by a deployed system, such as ongoing data processing for a platform like Facebook.

For (1), there is a crucial insight that is often missing: the unit of work for such people is not the program, but the execution. If you have a Makefile or a shell script or even a nicely source controlled program, you end up running small variations of it, with different parameters, and different input files. Very quickly you end up with hundreds of files, and no way of tracking what comes from which execution under what conditions. make doesn't help you with this. Workflow engines don't help you with this. I wrote a system some years ago when I was still in computational science to handle this situation (https://github.com/madhadron/bein), but I haven't updated it to Python 3, and I would like to use Python's reflection capabilities to capture the source code as well. It should probably be integrated with Jupyter at this point, too, but Jupyter was in its infancy when I did that.

For (2), there are systems like KNIME and Galaxy, and, crucially, they integrate with a LIMS (Laboratory Information Management System) which is the really important part. The workflow is the same, but it's provenance, tracking, and access control of all steps of the work that matters in that setting.

For (3), what you really want is a company wide DAG where individuals can add their own nodes and which handles schema matching as nodes are upgraded, invalidation of downstream data when upstream data is invalidated, backfills when you add a new node or when an upstream node is invalidated, and all the other upkeep tasks required at scale. I have yet to see a system that does this seriously, but I also haven't been paying attention recently.

For none of these is chaining together functions with error handling and reporting the limiting factor. It's just the first one that a programmer sees when looking at one of these domains.

It's not so much that Make is insufficient; there are a huge number of reasons to use something over make. However its true the difference between what you can implement with any reasonable amount of work and Make doesn't usually justify not using something simple that everyone understands (like Make). The next step up "worth making" involves hundreds of features that go deep into territory most people don't realize exists when they start re-inventing this particular wheel.

This library has a GPDR consent problem-- just putting it at the end of the README doesn't cut it. I wonder if this is the first open source library to violate GPDR.