Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

31–40 of 183 posts

Re: Show HN: Quilt – manage data like code

#31

Post by akarve »

purvis wrote: post #8
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Do you plan to curate the free plans? If not, how will you prevent abuse? (thinking in terms of both the service and its users)

Yes :) Happy to discuss in detail if you have specific types of abuse in mind. Our first line of defense is to get help from the community through downvoting useless content.

Re: Show HN: Quilt – manage data like code

#32

Post by dhbradshaw »

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This could be a big deal. Fantastic concept and great implementation.

Re: Show HN: Quilt – manage data like code

#33

Post by akarve »

tzm wrote: post #10

Thanks for releasing. Looks useful and aligned with several projects I've worked on. The first thing I looked for was a canonical package / resource specification in build.py. Any chance supporting Frictionless data resource spec for interop? https://specs.frictionlessdata.io/data-resource/

As Kevin mentioned we can extend support to frictionless (and are acceptign PRs on GitHub :). The thing we didn't love about frictionless is that it requires the user to fully specify the schema. We take a slightly more automated approach: https://docs.quiltdata.com/make-a-package.html https://docs.quiltdata.com/buildyml.html

I think we could generate a frictionless schema pretty easily...

Re: Show HN: Quilt – manage data like code

#34

Post by akarve »

fredcash25 wrote: post #25

How is this different/the same as the DAT project ( https://datproject.org/ and https://github.com/datproject )?

Dat is a distributed transport layer for raw data. Quilt is a centralized (your infrastructure or ours) transport and consumption layer for virtualized data. As such we'll be able to, for example, run efficient queries across all of Quilt, allow users to import data the same way (no data prep scripts) across a variety of platforms, etc.

Re: Show HN: Quilt – manage data like code

#35

Post by dpiers »

Wow - this looks like it would be really useful for us, and fits perfectly with our existing processes. I am building out the data analytics function on the Internal Audit team at Uber, and one of our challenges is that we have to pull and manage data from different business systems, and be able to track which version of a data set a report/analysis was run against.

It would be really cool if quilt could generate documentation for datasets, even if it was just column names/types. One of the issues we have is keeping track of all of the data "assets" people have pulled or created.

Re: Show HN: Quilt – manage data like code

#36

Post by akarve »

RangerScience wrote: post #14
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Also, if we want to produce our own data set for community consumption, what are your requirements, and what kind of payout could we expect?

To start with, we can add stars (pay with prestige). Getting more into science fiction--but very possible science fiction--we can put data on the blockchain and let people transact. The data owner would get the lion's share of the transaction.

Re: Show HN: Quilt – manage data like code

#37

Post by akarve »

dhbradshaw wrote: post #32
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This could be a big deal. Fantastic concept and great implementation.

Thanks. Where do you see it going in 2-5 years? Sometimes outsiders see things about the future that we haven't thought about :)

Re: Show HN: Quilt – manage data like code

#38

Post by purvis »

akarve wrote: post #31
purvis wrote: post #8

Earlier quoted context omitted.

Do you plan to curate the free plans? If not, how will you prevent abuse? (thinking in terms of both the service and its users)

Yes :) Happy to discuss in detail if you have specific types of abuse in mind. Our first line of defense is to get help from the community through downvoting useless content.

The main thing is how you are able to offer unlimited disk, when anyone can start uploading data? Some users might upload very large amounts of innocent data, others might simply abuse it as a sort of data backup system. More nefarious users might even try to distribute malware.

Re: Show HN: Quilt – manage data like code

#40

Post by kevinemoore »

dpiers wrote: post #35

Wow - this looks like it would be really useful for us, and fits perfectly with our existing processes. I am building out the data analytics function on the Internal Audit team at Uber, and one of our challenges is that we have to pull and manage data from different business systems, and be able to track which version of a data set a report/analysis was run against. It would be really cool if quilt could generate doc…

We're definitely planning to make column names and types more easily accessible and searchable. It'd be great to learn more about what information and metadata would be most useful to you. We also have an on-prem version we're rolling out with a couple of pilot customers in case that's helpful.
Post reply on HN