Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science Workflow: Overview and Challenges

cacm.acm.org

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Data Science Workflow: Overview and Challenges

#31

Post by paulgb »

earino wrote: post #19
paulgb wrote: post #17

Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.

I would love to know what you think about Domino Data Lab's solution. I feel pretty strongly about provenance tracking, especially being able to go: artifact experiment code / data environment, and our solution implements those linkages. Here's an older video describing it. https://www.dominodatalab.com/resources/data-science-experim... We now track things like git checkouts and dataset versions too.

It looks like the right approach for building a platform from scratch, the problem is that I am working with an existing in-house platform. Makefiles are so un-opinionated that they fit in quite nicely, all I really want is a more modern Make.

Re: Data Science Workflow: Overview and Challenges

#32

Post by paulgb »

in9 wrote: post #30
paulgb wrote: post #29

Earlier quoted context omitted.

I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the…

I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually

Yes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though.

(I don't actually use their template, I just use the directory layout as guidance)

Re: Data Science Workflow: Overview and Challenges

#34

Post by mring33621 »

in9 wrote: post #33

I am amazed that there is no mention of airflow in this thread.

Agreed. It seems to me that Apache Nifi (for provenance/replay) + jupyter/zepplin/r-studio (for exploring) + airflow (for final workflow) would be a good, open source starting point.

Re: Data Science Workflow: Overview and Challenges

#35

Post by in9 »

mring33621 wrote: post #34
in9 wrote: post #33

I am amazed that there is no mention of airflow in this thread.

Agreed. It seems to me that Apache Nifi (for provenance/replay) + jupyter/zepplin/r-studio (for exploring) + airflow (for final workflow) would be a good, open source starting point.

Never heard of Apache Nifi. Will dig into it! What is your basic use case for it?

Re: Data Science Workflow: Overview and Challenges

#36

Post by nl »

paulgb wrote: post #28
nl wrote: post #23

Earlier quoted context omitted.

Luigi works pretty well for workflow and versioning, if not complete provenance.

Luigi doesn't support incremental builds, does it? I think that was my main contention the last time I looked into it.

What do you mean by incremental builds?

It certainly lets you have some features pre-built and it will check for them and use them.

Re: Data Science Workflow: Overview and Challenges

#37

Post by pea »

This is really interesting. Firstly, I wonder how fundamental notebooks have becoming to the preparation phase in the last couple of years (to avoid https://cacm.acm.org/system/assets/0001/3683/long-filenames....).

It also strikes me that a lot of these problems in dissemination have been solved to a large degree in software development. I guess 'writing code everyday' and 'knowing how to architect systems and use software development principles' are getting increasingly less mutually exclusive, which lends itself well to further abstractions.

Distribution is one of the things we deal with most frequently at https://nstack.com. We let data scientists deploy their models into bit-for-bit reproducible modules which run in the cloud and can be composed with various data sources. Because these are semantically versioned and packaged on the server as bit-for-bit reproducible builds, this solves the fact that "one's own operating system and software inevitably get upgraded in some incompatible manner such that the original code no longer runs". Our background was in writing programming languages, so we took a lot of the ideas of type-safety and composition and applied them to this problem.

Actually the most common way we see dissemination solved in enterprises is folks writing back to the warehouse and using Looker / Tableau to visualise, with a workflow like:

    import Redshift:0.1.0 as Redshift
    import CustomerModel:0.0.2 as Classifier

    def workflow = Schedule { interval = "Daily"} | Redshift.query("cookies.sql") | Classifier.predict | Redshift
Looking forward to jumping into his http://pgbovine.net/cde.html project. We explored a ton of solutions here (including Nix), but ended up settling on btrfs.

Re: Data Science Workflow: Overview and Challenges

#38

Post by dwhitena »

paulgb wrote: post #17

Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.

I will also throw in my (biased, as I work on the project) suggestion to take a look at Pachyderm (http://pachyderm.io/). It is open source, language agnostic, and distributed. Plus it automatically tracks the provenance of all of your data pipelines, regardless of language or parallelism over your data.

Basically you set up data pipelines, where the input/output of each stage is versioned (like "git for data"). That way you have versioned sets of your data (e.g., training data), but you also can know things like exactly what model was used to produce which result, what data was used to train that particular model, what transformations occurred on that training set, etc.

Things like Airflow and Luigi are, no doubt, useful for data pipelining and some workflows (depending on what language you are working with). However, by combining pipelining and data versioning in a unified way, Pachyderm naturally lets you handle provenance of complicated pipelines, have exact reproducibility, and even do interesting things like incremental processing.

Re: Data Science Workflow: Overview and Challenges

#39

Post by jaz46 »

pea wrote: post #37

This is really interesting. Firstly, I wonder how fundamental notebooks have becoming to the preparation phase in the last couple of years (to avoid https://cacm.acm.org/system/assets/0001/3683/long-filenames.... ). It also strikes me that a lot of these problems in dissemination have been solved to a large degree in software development. I guess 'writing code everyday' and 'knowing how to architect systems and use s…

>It also strikes me that a lot of these problems in dissemination have been solved to a large degree in software development.

Yeah, this is exactly how we feel too! Git solves this so well for software development which is why Pachyderm (http://pachyderm.io/pfs.html) build a distributed file system with those same principles in mind.

Post reply on HN