Any relation to https://github.com/QuiltProject ?
No, no relation.
Show HN: Quilt – manage data like code
91–100 of 183 posts
Re: Show HN: Quilt – manage data like code
#92Any thoughts on adding DOIs? It's a complex subject wrt versioning, in particular (new DOI per version? How to keep track?). It would help tremendously with the academic community; for the bean counting.
The package name + hash is an implicit DOI. What if we added web support for it so that users could https://quiltdata.com/packages/USER/PKG?doi=SOME_HASH ?
This is more than adequate. DOIs are simply redirects. It's up to the data owner to point the DOI at whatever resource contains the data. If a DOI is registered, it can be pointed to the quilt URL.
You can take it a step further and either integrate with a DOI provider or become one yourself and integrate the registration process within your api or create command line tools.
Re: Show HN: Quilt – manage data like code
#93The workflow I would have imagined for versioning data is:
1) Load original data from source into quilt
2) Do transformation
3) Commit transformations to quilt, with commit message
4) Run experiment
5) Do new transformations
6) Commit to quilt
7) Run experiment
Rinse and repeat.
Looking at the video and documentations, this is not emphasised at all, suggesting that edits to data should be saved as a new package.
Re: Show HN: Quilt – manage data like code
#94Earlier quoted context omitted.
No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.
Huh, that's a pretty good setup for you then! I might mess around and see about writing an R package/interface, because this looks very useful.
That would be awesome! We're happy to help, but we only know a little bit of R. We've been looking at Sparklyr in case it might help read Parquet into R.
Re: Show HN: Quilt – manage data like code
#95It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…
I just joined a company that solves this very problem: https://www.delphix.com/ What you described is literally what we offer -- the ability to go backwards and forwards in time and create branches, bookmarks, and refreshes with very little overhead. [Edit: This is all for databases that by default, as you noted, don't do any of this stuff.] We aren't (yet) super well known, but we've got a number of things in flight…
How does it work? Is it filesystem snapshots? I know it's unreasonable to expect, but do you support forking/merging? What about incremental or logical replication?
Re: Show HN: Quilt – manage data like code
#96The workflow I would have imagined for versioning data is: 1) Load original data from source into quilt 2) Do transformation 3) Commit transformations to quilt, with commit message 4) Run experiment 5) Do new transformations 6) Commit to quilt 7) Run experiment Rinse and repeat. Looking at the video and documentations, this is not emphasised at all, suggesting that edits to data should be saved as a new package.
We definitely imagine saving edits as creating new versions of the same package. The most common pattern we've heard is adding more dataframes or files to a package as new results are generated. But, we can certainly imagine other transformations.
Re: Show HN: Quilt – manage data like code
#97Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…
Could you tell us how this compares with Synapse (https://www.synapse.org/)?. They've been doing this for a long time and have a large presence in computational biology.
Re: Show HN: Quilt – manage data like code
#98Earlier quoted context omitted.
Is there planned support for any other languages than Python?
Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.
I'd definitely be interested in looking into this but I'm on an R team. Didn't see anything on how to contribute in another language or an API to hit. Will be there be information soon?
Re: Show HN: Quilt – manage data like code
#99It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…
And most databases can't tell you who put that bad data into the database.
Nearly every place I've worked at we've had to try to create our own version history and audit information. This one of the examples I use when people start talking about how we have a shortage of engineers.
No, we have a shortage of good tools. Nobody should have been writing their own data versioning code in 2007, let alone 2017.
Re: Show HN: Quilt – manage data like code
#100Earlier quoted context omitted.
This looks pretty cool. Am I right to understand that this is providing a virtualized filesystem interface that dynamically loads the slices of data actually being accessed (but through a direct API rather than something generic like FUSE)? Unrelated I found a typo on your blog. Search for "seriailize" on your " Manage data like source code " post.
As to your slicing question, yes. Data is lazily loaded. With Parquet as our data store we can do even more (but haven't yet): load only the columns referenced.
If you're talking about lazy loading of data, for your R implementation you might want to (if you're not already) look into creating a custom dplyr backend that only loads data when needed (similar to the dplyr SQL backends).