Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

51–60 of 183 posts

Re: Show HN: Quilt – manage data like code

#51

Any thoughts on adding DOIs? It's a complex subject wrt versioning, in particular (new DOI per version? How to keep track?). It would help tremendously with the academic community; for the bean counting.

Good suggestion! From what we've heard from academic users, they'll want a DOI for a specific version, e.g., data from a particular paper or journal article. Any thing else we should watch out for?

Re: Show HN: Quilt – manage data like code

#52
post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This is a really impressive project. Do you think you would add data cleaning commands? How do you think you would handle datasets that are only available to academics or other restrictions? On your front page you should have the owner/dataset name instead of just the dataset name so I wouldn't have to click through to find the owner name.

Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any "out of the box" cleaning functions you'd like? In the spirit of dplyr? We've looked at e.g. scikit feature for normalization, 1-of-n encoding, etc.

Re: Show HN: Quilt – manage data like code

#53
post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

are you aware of datapackages [1]? do you plan on opensourcing the backend components?

[1]: https://github.com/frictionlessdata

Re: Show HN: Quilt – manage data like code

#54
I am a scientist who sometimes publishes data sets with academic papers and this looks super useful both as a tool and as a potential publishing best practice. Currently doing away with HDF5 and figshare. One necessary future for academia would be to be able to assign a DOI to a given version of data. Is it feasible for quilt to have such a feature?

Re: Show HN: Quilt – manage data like code

#55
post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

Re: Show HN: Quilt – manage data like code

#56
post #13

Earlier quoted context omitted.

Any way your web page could be made viewable without requiring Javascript? I disable it in my web browsers because of security and privacy concerns, and it would be great if I could just read about your project without it.

Ah, that would be useful. In the meantime our docs are browsable without JS https://docs.quiltdata.com/ . The articles linked at the bottom of the docs landing page are also legible without JS. Unfortunately the docs menu is not available without JS. I will look into fixing these issues. Email is feedback at quiltdata dot io if you have questions :)

The menu on https://docs.quiltdata.com/ isn't visible if you have JS disabled. I had to disable CSS to be able to navigate.

Re: Show HN: Quilt – manage data like code

#60
post #29

Earlier quoted context omitted.

This is pretty neat! Is there an API or anything, so we can write support for other languages?

Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.

Ruby and Go, personally.
Post reply on HN