Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D6tflow: Python library for building data science workflows

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: D6tflow: Python library for building data science workflows

#41

Post by mlthoughts2018 »

anigbrowl wrote: post #40
mlthoughts2018 wrote: post #13

I feel so frustrated by the emergence of these things and the constant attempt to brand them or stylize them towards data science. There are just engineering projects. There are not any other things. For some engineering projects, you need to support dashboard-like, interactive interfaces that depend on data assets or other assets (like a database connection, a config file, a static representation of a statistical mo…

But I don't want to learn 50 different tools that are all best-in-class. My use case is social media analysis of specific communities with fairly limited resources, so every hour that I spend on tooling is time not spent observing my subjects. You're not wrong about the value of all the different tools you mention, but I think overlooking the integration and maintenance costs that a specialty tool can reduce, at the…

Learning the time tested tools almost always involves spending less time setting up / reading tutorials / etc. The time sink of betting the farm on latest and greatest data science frameworks is often gigantic and gets worse over time.

Re: D6tflow: Python library for building data science workflows

#42

Post by theblackcat1002 »

The reasons I choose not to use any pre-existing workflow are complexity and extra compute overhead. I think there's a need for micro workflow where it's simple enough you can setup a simple production workflow in a single python script without requiring a ton of heavy package. d6tflow seems light enough for me, using luigi and two dataframe libraries.

Re: D6tflow: Python library for building data science workflows

#43

Post by sorryforthethro »

SebiH wrote: post #15

> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…

Also interesting to see what happens when such a library becomes nested within a more popular library. Disabling should still work, but fewer people would be aware.

Re: D6tflow: Python library for building data science workflows

#45

Post by ematvey »

m0zg wrote: post #37
madhadron wrote: post #25

Earlier quoted context omitted.

It's easy to make the case that make is insufficient. There are three very distinct domains being conflated here: 1. The day to day experimentation and iteration by a computational researcher. 2. The repeated execution of a workflow on different data sets submitted by different people, such as in a clinical testing lab. 2. The ongoing processing of a stream of data by a deployed system, such as ongoing data processin…

> what you really want is a company wide DAG where individuals > can add their own nodes and which handles schema > matching as nodes are upgraded, invalidation > of downstream data when upstream data is invalidated I've never seen this work in practice, and doubt it can work, due to the complexities involved.

It really helped in our case. We have a team of 10+ researchers who alse ship code in production. They were repeatedly running into a problem where they recompute same data in runtime, or reinvent the wheel because they didn’t know somebody already computed that datum. I end up writing a small single-process (for now) workflow engine running a “company-wide DAG” of reusable data processing nodes (all derived from user-submitted input + models). Now it is much easier for individuals to contribute + much easier to optimize pipelines separately. I might open source it some time soon.

Re: D6tflow: Python library for building data science workflows

#46

Post by madhadron »

m0zg wrote: post #37
madhadron wrote: post #25

Earlier quoted context omitted.

It's easy to make the case that make is insufficient. There are three very distinct domains being conflated here: 1. The day to day experimentation and iteration by a computational researcher. 2. The repeated execution of a workflow on different data sets submitted by different people, such as in a clinical testing lab. 2. The ongoing processing of a stream of data by a deployed system, such as ongoing data processin…

> what you really want is a company wide DAG where individuals > can add their own nodes and which handles schema > matching as nodes are upgraded, invalidation > of downstream data when upstream data is invalidated I've never seen this work in practice, and doubt it can work, due to the complexities involved.

It's what is done de facto by large enough groups anyway. They just have to kludge tooling together for it.

Re: D6tflow: Python library for building data science workflows

#47

Post by prepend »

SebiH wrote: post #15

> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…

I thought this was a neat package and would have tested it. I won’t install or use it because of this.

I don’t like packages that require external access to function. I understand the business mode and think there are clear ways to do this (plotly and graphistry come to mind), but I don’t think the benefit outweighs the downsides to use these types of libraries.

At least this can be easily refactored out, plotly and graphistry don’t really function well without the api calls. Plotly offline exists, but trying to keep track of features between the two is a pain. And the reasoning given for the api (massive scale conpute) could be easily abstracted for local mode if they wanted.

Re: D6tflow: Python library for building data science workflows

#49

Post by marmaduke »

madhadron wrote: post #25
marmaduke wrote: post #18

I recently read that there are over 100 workflow engines [1], it is still often difficult to make the case that Make is insufficient [1] https://vatlab.github.io/blog/post/sos-workflow-engine/

It's easy to make the case that make is insufficient. There are three very distinct domains being conflated here: 1. The day to day experimentation and iteration by a computational researcher. 2. The repeated execution of a workflow on different data sets submitted by different people, such as in a clinical testing lab. 2. The ongoing processing of a stream of data by a deployed system, such as ongoing data processin…

Ok but none of the those situations can be addressed with zero budget. If you’ve got those problems to solve then you usually ha e a budget to build/buy an appropriate tool.

At the other end of the spectrum you have every small team with some data analysis steps producing their own workflow engine when Make would be just fine.

I agree however the streaming case is particularly poor, but consider that paired with an appropriate fuse file system Make can address most use cases.

Re: D6tflow: Python library for building data science workflows

#50

Post by nicolaskruchten »

prepend wrote: post #47
SebiH wrote: post #15

> We have put a lot of effort into making this library useful to you. To help us make this library even better, it collects ANONYMOUS error messages and usage statistics. See d6tcollect for details including how to disable collection. Collection is asynchronous and doesn't impact your code in any way. That seems really out of place. I'm somewhat used to automatic data collection from applications, but automatic data…

I thought this was a neat package and would have tested it. I won’t install or use it because of this. I don’t like packages that require external access to function. I understand the business mode and think there are clear ways to do this (plotly and graphistry come to mind), but I don’t think the benefit outweighs the downsides to use these types of libraries. At least this can be easily refactored out, plotly and…

Actually, Plotly offline works exactly the same was as Plotly online: same code runs on both ends, total feature parity from Python :)
Post reply on HN