Earlier quoted context omitted.
Just use AWS S3 (or similar) and shell scripts. My team uses a git repository named something like "data-packages", which is nothing but a collection of shell scripts with the name .sh, that perform the necessary download and extraction steps to get a dataset from S3. Data sets are immutable by convention, so any changes to a data set requires you to provide a totally new shell script. That script could download an o…
Interesting thoughts. Quilt has a ways to grow. You correctly point out that, in some cases, S3 is lighter weight. You'll see future versions of Quilt get lighter, and offer more S3-like "just store this" functionality. In its next minor revision, Quilt simplifies point updates (i.e. it will be possible to update a single training example without materializing the entire package). That said, there are a few areas whe…
I actually disagree with this. In a Python-like “consenting adults” philosophy, I think it’s worse to spend engineering effort to guarantee immutability rather than to trust people not to and just have a reasonable system of backups.
Immutability by convention is 99.99999999% as good as enforced immutability for this particular type of task, and there’s even less risk with a good backup strategy to fall back if there is an accident.
Change history and access auditing are super easy on S3, as is fine-grained access control. With immutability by convention, change history is just git history, and you can customize access groups on a file-by-file basis if you want. You could also instrument logging in the shell scripts themselves if you really want, and I’m not convinced that’s worse than a third party doing it, especially if your logging backend is prone to change, or you wantbyo pipe stuff to Grafana, etc., which are quite common needs.
Querying and filtering are separate post-processing tasks. They should be expressed as source code that mutates a data set after downloading a local copy, and in fact version controlling any data cleaning, post-processing, etc., should be kept completely separate from the management of a data package. They are logically hugely different parts of the process. Slicing data ought to be up to the individual developer or researcher, to choose their tools, to optimize, etc. Version control of that source code is the right way to make that part of the work reproducible, not trying to tie custom treatments into a version of a data package.
Your points about dedup and deserialization are good ones. I can imagine problem cases for a simple script approach, but I can also say even for gigantic in-house image data sets, creating multiple slightly different materialized copies has rarely been an issue.