Do a lot of these "data quality" problems go away if you were simply to use a relational DB, and related best practices, like table versioning, PKs, using a last_updated timestamp for each row, etc.?
Data Science Workflow: Overview and Challenges
21–30 of 39 posts
Re: Data Science Workflow: Overview and Challenges
#22Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository.
That is my goto combinations for bioinformatics workflows that don't need to run on a cluster. (A Makefile variant of mine is my go-to for clusters). I've experimented with Docker workflows, which while in theory are nice, you'll always require some sort of organizing script to document the analysis. (Docker is still nice for capturing program versioning/configuration though!)
Re: Data Science Workflow: Overview and Challenges
#23Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
Luigi works pretty well for workflow and versioning, if not complete provenance.
Re: Data Science Workflow: Overview and Challenges
#24Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
Luigi works pretty well for workflow and versioning, if not complete provenance.
SciLuigi adds a simple level of audit logging to Luigi [1] (See the AuditTrailHelpers mixin in https://github.com/pharmbio/sciluigi/blob/master/sciluigi/au... for details), together with an API that makes it much easier to work do changes to the workflow dependency graph, when you're in development/exploratory phase.
Personally moved to improve on the ideas in SciLuigi, but replacing Luigi with Go's concurrency primitives, in SciPipe [2].
In SciPipe, an accompanying ".audit.json" file, containing the full history of commands and parameters used to create every single output file of the workflow (even intermediate ones), to help keep track of how each file was produced.
Among lightweight solutions, the popular ones these days seem to be NextFlow [3], Snakemake [4], BPipe [5] and others. You'd really have to check out the "awesome pipelines" [6] list, to get any kind of overview.
When more infrastructure setup is feasible, I think PachyDerm [7] (mentioned elsewhere in the thread) looks really powerful with its "Git for data" approach. Something I'd wish to use as an overarching solution within which to run my SciPipe workflows.
[1] https://github.com/pharmbio/sciluigi [2] http://scipipe.org [3] http://nextflow.io [4] https://bitbucket.org/snakemake/snakemake/wiki/Home [5] http://bpipe.org [6] https://github.com/pditommaso/awesome-pipeline [7] http://pachyderm.io
Re: Data Science Workflow: Overview and Challenges
#25Philips CDE tool [1] looks like such a beautiful thing!
> "Prepend any set of Linux commands with the "cde" binary, and CDE will run them and automatically package up all files (e.g., executables, libraries, plug-ins, config/data files) accessed during execution."
Then, put this in a Docker if you want, but so awesome not be required to do that!
I think more orthogonality is needed, and less fluff and accidental complexity in all its forms, including "buzzword" tech stacks.
Re: Data Science Workflow: Overview and Challenges
#26Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository. That is my goto combinations f…
You could use Git-LFS for managing huge data blobs with Git?
Re: Data Science Workflow: Overview and Challenges
#27Shameless plug: looking specifically at the Dissemination Phase part, I can't help but thinking he would love Code Ocean ( https://codeocean.com ). It addresses each of his points: - distribute software (execute with a single click) - reproduce results (published code & data - yours and others' - is archived with its environment) - collaborate with colleagues (built-in to the system) (Disclaimer: I work at Code Ocean…
Suggestion: Keep an eye on Go as an emerging data science language (See http://gopherdata.io/ for more info)
Re: Data Science Workflow: Overview and Challenges
#28Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
Luigi works pretty well for workflow and versioning, if not complete provenance.
Luigi doesn't support incremental builds, does it? I think that was my main contention the last time I looked into it.
Re: Data Science Workflow: Overview and Challenges
#29Nearly four years after this was written, there is still no satisfying solution to providence tracking in a data science workflow. The few solutions that do exist tend to be too opinionated to catch on. I've gotten a lot of mileage out of hacking some provenance tracking into a plain-old-Makefile workflow, but it's hard to balance human-readable makefiles and the amount of hackery it takes.
I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository. That is my goto combinations f…
I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/
I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.
Re: Data Science Workflow: Overview and Challenges
#30Earlier quoted context omitted.
I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository. That is my goto combinations f…
I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the…
I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually