Show HN: Quilt – manage data like code • Hacker News Board

Re: Show HN: Quilt – manage data like code

#21

Post by kevinemoore » Fri, Jul 14, 2017, 5:51 PM UTC

zitterbewegung wrote: post #5

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This is a really impressive project. Do you think you would add data cleaning commands? How do you think you would handle datasets that are only available to academics or other restrictions? On your front page you should have the owner/dataset name instead of just the dataset name so I wouldn't have to click through to find the owner name.

We've been thinking of making a Python build file to make it easier add richer transformations than the simple file->DataFrame that runs by default. We'd love suggestions and input on the types of transformations that would be most useful.

Re: Show HN: Quilt – manage data like code

#22

Post by masklinn » Fri, Jul 14, 2017, 5:52 PM UTC

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Re: Show HN: Quilt – manage data like code

#23

Post by adriancooney » Fri, Jul 14, 2017, 5:52 PM UTC

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Is there planned support for any other languages than Python?

Re: Show HN: Quilt – manage data like code

#24

Post by akarve » Fri, Jul 14, 2017, 5:53 PM UTC

koolba wrote: post #7

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This looks pretty cool. Am I right to understand that this is providing a virtualized filesystem interface that dynamically loads the slices of data actually being accessed (but through a direct API rather than something generic like FUSE)? Unrelated I found a typo on your blog. Search for "seriailize" on your " Manage data like source code " post.

Typo fixed. Thanks :)

Re: Show HN: Quilt – manage data like code

#25

Post by fredcash25 » Fri, Jul 14, 2017, 5:53 PM UTC

How is this different/the same as the DAT project (https://datproject.org/ and https://github.com/datproject)?

Re: Show HN: Quilt – manage data like code

#26

Post by kevinemoore » Fri, Jul 14, 2017, 5:53 PM UTC

tzm wrote: post #10

Thanks for releasing. Looks useful and aligned with several projects I've worked on. The first thing I looked for was a canonical package / resource specification in build.py. Any chance supporting Frictionless data resource spec for interop? https://specs.frictionlessdata.io/data-resource/

We've definitely been looking at that! Are you using frictionless data packages now?

Re: Show HN: Quilt – manage data like code

#27

Post by thenipper » Fri, Jul 14, 2017, 5:55 PM UTC

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

What's your privacy policy for private data?

Re: Show HN: Quilt – manage data like code

#28

Post by kevinemoore » Fri, Jul 14, 2017, 5:56 PM UTC

adriancooney wrote: post #23

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Is there planned support for any other languages than Python?

Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.

Re: Show HN: Quilt – manage data like code

#29

Post by akarve » Fri, Jul 14, 2017, 5:57 PM UTC

RangerScience wrote: post #6

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This is pretty neat! Is there an API or anything, so we can write support for other languages?

Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.

Re: Show HN: Quilt – manage data like code

#30

Post by akarve » Fri, Jul 14, 2017, 6:00 PM UTC

thenipper wrote: post #27

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

What's your privacy policy for private data?

Only the owner and designated collaborators can view it. We offer private data in the cloud (S3) and are rolling out on-premise in case you want to run Quilt on your own infrastructure, in which case you control blob storage. Does that answer your question?